#Tidiness facts code#

tell tidy I want it to preserve the original meta charset value, so I can run xpath over the clean xml (I don't like this idea much).

So what I'd need is one of the following: Anything simpler, like a regex match for example is just doomed to fail sooner or later. If I didn't want to use xpath then I'd have to manually parse the html before tidying, deal with invalid input etc, so in short duplicate what tidy already does. See my problem? After I clean the html it's ready to be fed to the xpath library, but because tidy overwrites the charset field then my xpath query always returns utf-8, no matter what. I can easily retrieve the first meta tag using xpath (after all that's what my program does), but before I can do that html needs to be cleaned and become valid xml. In this case the server just reports Content-Type: text/html so what matters is the meta tag. This is confusing - charset=windows-1252 in the input file is correctīut tidy is saying it's that sounds like a good solution, and to be honest that's what I set out to do at the start. $ tidy -show-meta-change yes -input-encoding win1252 This is nice - the user has a pretty good hint why tidy complains 'text/html charset=windows-1252' replaced Line 17 column 1 - Info: attribute 'content', incorrect value Line 5738 column 43 - Warning: replacing invalid UTF-8 bytes (char. Line 4404 column 31 - Warning: replacing invalid UTF-8 bytes (char. Line 2422 column 53 - Warning: replacing invalid UTF-8 bytes (char. Line 2422 column 31 - Warning: replacing invalid UTF-8 bytes (char. Line 1254 column 61 - Warning: replacing invalid UTF-8 bytes (char. Line 151 column 81 - Warning: replacing invalid UTF-8 bytes (char. Line 151 column 36 - Warning: replacing invalid UTF-8 bytes (char. Line 125 column 33 - Warning: replacing invalid UTF-8 bytes (char. Line 119 column 33 - Warning: replacing invalid UTF-8 bytes (char. Line 1 column 1 - Warning: missing declaration $ tidy -show-meta-change yes x.htm >| x.html Geoff, the '-show-meta-change yes' option seems not quite right: The HTTP header overrides all in-document encoding declarations in

In particular, it is important to note that the encoding declared in

It seems the more correct way is to first look for the encoding when I'm trying to figure out if I'm doing something wrong myself, it might be this is indeed the expected behaviour, in which case my question is how can I tell what the original encoding was so I can invoke iconv() correctly? Thanks. If it's false, then I get an html file that claims to be utf8 but that contains non-utf8 sequences for the accented characters.

#Tidiness facts code#

The page linked in my first post only produces a correct utf8 output if that src_charset in my code is true and the if branch is taken. This is how I set up tidy, then if you look at line 108 if the user specified a charset explicitly then I apply it, otherwise I just go with tidy's default. Destination is always utf8, so that's easy, but what about input? In my test, the meta charset tag from tidy always says utf8, even when that's not the case, because as you pointed out, tidy does not convert characters unless you explicitly ask it to do so. It's ok if I have to use iconv for that, but in that case the problem is I need to tell iconv about the source charset and the destination charset. What I want to do is normalise my input so that my code only ever deals with utf8. I'm a bit confused about what I should do. maybe there is a bug here, but not seeing it yet. If not, please advise the character code(s) that are causing a problem, and the config used. With the latter got entities like – for 0x96, and é for 0圎9. I had no problems with the online page you linked to, using a config of -raw, or -win1252. note the default chosen pairs for in and out.

See how tidy internally handles this, AdjustCharEncoding. also see show-meta-change & add-meta-charset.Īs you have shown, in your own code, you need to set input, but you must also set output, else it will default to utf8. Tidy `ignores` the meta charset in the doc, on input. Tidy `default` in, and out, encoding is UTF8.