Like most dance-halls, it was accused of being a haunt of prostitutes/women looking for unattached amusement - much the same thing, according to nineteenth century moralisers. You can find some more stuff about it on my site here where it generally comes out rather well, as opposed to the 'better class' establishments, such as the Holborn Casino, which does seem to have been a notorious pick-up joint.
This is what reminded me of it - a great account I've just stumbled upon, from its declining years:
If you have ever lost yourself in Soho, or been to the Royalty Theatre, you have probably seen this somewhat seedy-looking academy - and what it looked outside, it was in - for it was, without exception, I should think, the seediest, shabbiest, dirtiest, "tumble-downiest" place of amusement you could find in London. The price of admission stamped it in my mind. Fancy, eightpence. What a miserable sum! Sixpence sounds much more respectable. Once inside the turnstile, at which a melancholy man, who always had a glass of rum and water before him, presided, and up the staircase, where a spotted mirror or two, and some dirty, cracked, plaster statues kept up the seedy idea, you came to the dancing-room, a large bare apartment with everything in it in the way of decoration utterly gone to the bad. One end of it was a gallery where the "music" sat. Ye gods! What a band was that. "Seedy" to its very core - with its cornet always cracked and its other instruments either imbecile or drunk. There were two seedy waiters, too, most weak-kneed and flat-footed of their race; who - no liquor licence being attached to the place - were kept running to and fro, between it and the proximate public at the next turning. As to the usual audience, it well matched its surroundings. There was none of the flaunt and glare of the Argyll or the Holborn about it. Caldwell was largely supposed by that class of girl called, I believe, in select circles, "dolly-mops"; a sort of uninteresting and seedy edition of the Parisian grisette. Ballet girls out of an engagement and "slaveys" out for the night also patronised it; and the men who went there were almost, without exception, snobs or cads. Such is my idea of a place the Observer saw fit to gush about in a most sickening way last Sunday. The Holborn is at any rate lively, and you get good music, and something pretty to look at - but Caldwell's, faugh! the place was as dreary as a gospel-hall. I think it a good job is has gone.
What the young men who used to go there in the day and take private lessons in dancing will do without Caldwell's I don't know, and don't care; though I believe Mr. Bland and his daughters and Miss Leonora Geary are still ready to take them in hand if they like. At Caldwell's, I understand, the mysteries of the trois temps and the galop were imparted by a superannuated ballet mistress, who was too old and fat to arouse amongst her pupils anything like a wish to intersperse the learning of their steps with amatory amusement, and the ballet girls provided as "lady partners" for the more proficient were, I believe, always very lean and ugly, for obvious reasons. Poor girls! what they must have suffered. I can fancy nothing worse than to be the partner of an awkward clumsy lout who is learning to waltz, unless indeed it is to be the wife of a man like Mr. Ruskin, who is wholly wedded to his art. It is cruelty to allow girls to be roughly and hardly used. Idiots who cannot dance should buy a sixpenny "Guide to the Ball Room" and practice at home with a chair, then they can hurt no one but thermselves.
Sporting Times, September 30, 1871
