BenBE's humble thoughts Thoughts the world doesn't need yet …

03.05.2009

Again some parser tweaking

Filed under: GeSHi — Schlagwörter: , , , , , , — BenBE @ 13:03:02

After 1.0.8.4rc1 contained a somehow major update on the parser (I reordered the processing of keywords, numbers and regexps) the next Release Candidate (1.0.8.4rc2) will contain yet another minor reordering as I noticed a small problem with Delphi. As always I’m hereby asking you to test this new release candidate once it is out and report any issues you find to me. If you don’t want to wait that long, you can always have a look at the trunk where these changes already have been applied. (more…)

Flattr this!

11.04.2009

GeSHi now without language file warnings

Filed under: GeSHi — Schlagwörter: , , — BenBE @ 13:30:14

It has been a lot of work and due to limited delight in code monkey work the final work had been delayed for several months; but now it finally has been completed: The language file validation script doesn’t show any warnings when validating the language file directory! (more…)

Flattr this!

26.03.2009

Roadmap for GeSHi 1.0.8.4

Filed under: GeSHi — Schlagwörter: , , , — BenBE @ 23:44:34

This March has been a busy month and I nearly missed to celebrate one year of working on GeSHi as head of developement. It has been an awesome time, although a bit complicated sometimes, hasty in other moments, but overall it was a pleasure to work on such a great project while learning many new things. (more…)

Flattr this!

01.03.2009

GeSHi 1.0.8.3 a bit later

Filed under: GeSHi — Schlagwörter: , — BenBE @ 13:32:39

Due to heavy workload on my side, including moving to a new server the upcoming 1.0.8.3 release of GeSHi will probably come out a bit later. Current estimation is mid-March but I’m trying to do my best to get it out as soon as possible.

This has become necessary as my old provider had the brilliant idea of requesting more money for as little performance as you are experiencing right now on this server. That’s why I’m switching to a new server during the course of the next 14 days. I hope moving the DNS goes as smoothly as the last time so you won’t have to wait for the site to be back up again.

During this time mails to my server will probably arrive delayed but I’m trying to keep delays as little as necessary.

And now on to a second thing I got notified of: Recent versions of libpcre seem to have problems with long matches inside repeating alternation groups. For more detaails see the bug report from Milian. Until they fix this issue upstream you might want to downgrade to an earlier version of libpcre (on Win libpcre 7.3 2007-08-28 takes long, but doesn’t crash) or comment out line 1080 and 1081 in the PHP language file (but you’ll loose the more precise PHP block detecktion).

This much for now. I’m open for your comments!

Flattr this!

17.01.2009

Probably only few new features this time

Filed under: GeSHi — BenBE @ 19:01:44

How time passes by. It’s already middle of January and there hasn’t been much work on GeSHi yet. Not that there was no time (which wasn’t, but that’s some other story) but there has only been little input so far thus no real changes were needed.

Well, actually there was some input in the form of some mails that pointed out some glitches I somehow missed in the previous release. First of all there was some kinda ugly bug that I got informed about multiple times: In the AutoIt language file there was a whole keyword group that made the parser go berserk. I couldn’t quite find the reason for this as the parser normally should allow underscores as first symbols of a keyword (if they aren’t set as symbols), but who knows. A short parser control that matches the first underscore if not preceeded by some alphanumeric character and the removal of the underscore from the actual keyword group did the job. And since this ment some major reformatting of the list: Don’t mention AutoIt anytime soon: I hate this file for now!

The other problems were minor ones. All dialects of C and C++ should now properly support multiline-continued preprocessor directives. This isn’t quite a new feature, but some point I missed, when I initially added multiline continuation in C for single line comments in some of the previous releases.

And as we already are at missing stuff: In the PHP language file I missed to escape the backslash properly, thus double-quoted strings ending in an escaped backslash weren’t properly ended as the ending double-quote was escaped instead of the escaped backslash.

So only keeps waiting for some more input and feedback for the remaining time till the release which probably will be in mid-February. Until then I’m not sure if I can invest a lot of time since exams at University call for attention. But well, time for integrating your patches and language files will of course be enough. Overall 1.0.8.3 will be mostly a bugfix release. IDK if I will issue some Release Candidates beforehand as there barely are changes requiring one.

So let’s see!

Flattr this!

09.01.2009

Highlighting Perl in all its oddity

Filed under: GeSHi — Schlagwörter: , — BenBE @ 00:02:34

Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /var/www/virtual/benny-baumann.de/blog/htdocs/wp-content/plugins/wp-syntax/wp-syntax.php on line 380

Warning: WP_Syntax::substituteToken(): Argument #1 ($match) must be passed by reference, value given in /var/www/virtual/benny-baumann.de/blog/htdocs/wp-content/plugins/wp-syntax/wp-syntax.php on line 380

Following up on a discussion on the GeSHi-devel mailing list I got some interesting testcases for testing Perl in all its oddity.

So lets get things startet: Open up the testcase, and see: The CSS is not fully precise on some stuff (some classes in GeSHi can be used for different things like .co5 for perl is used for predefined variables). So I did a quick editing of the CSS stylesheet with the Web Developer Toolbar to mark everything that was of specific classes (as expected) in a different color than white to see if it actually is highlighted (which most stuff was).

Adding the missing stuff into the language file I got an updated version for Perl that can be found as revision 1975 in the GeSHi SVN repository for testing. Things I didn’t include there are file descriptors as they should already be highlighted (even if <> and <test> are marked as different things.

But in one thing the testcase wasn’t right: Heredocs As well as Regular expressions are supported by GeSHi, see:

$test = <<HEREIAM;
Some text to show this is a heredoc part
HEREIAM;
$match =~ /te[sx]t/g;
$replace =~ s/x/s/i;

Just to be complete on this. Oh, and the

< <

bug is a problem with my WordPress. The demo page does the Heredoc correctly, for those that don’t believe me 😉

Now I’m only lacking this little testcase as a file for inclusion into the Code Repository of GeSHi and I can declare this „done“ …

Update: Revision 1977 contains some important updates on the rev1975\r1974 version of this file: I missed some things due to some issue I had with encoding $ inside of regexps (I missed the escapes PHP needed inside the string). I also added the few remaining cases that the previous version didn’t yet.

What will not be highlighted correctly are bacreferences to numbers (and I thought C was odd …) and nested Heredocs (i.e. Heredocs with two or more starters on one line).

But given the current state this should somewhat resemble most of what even most insane Perl programmers need when puking their code …

Flattr this!

24.12.2008

No Winners in the GCC

Filed under: GeSHi — Schlagwörter: , , — BenBE @ 17:28:42

A few months ago I announced the GeSHi Contributions Contest where I asked everyone to send in their contributions for other little, useful scripts that everybody could use and that utilize GeSHi in some innovative way.

Thereafter was little to nothing in response even though Milian and me, as well as some others, advertised this contest with our limited resources and time as well as we could. In particular I heard about one case where some professor considered this contest could be nice, but didn’t enter because of lack of time (the timeline was to short for his students).

Therefore I’d be grateful to get some response to this contest and things that could be done better, more and so one.

I’m really looking forward to your feedback.

BenBE.

Flattr this!

12.12.2008

Bug Fixing – The End of the Story

Filed under: GeSHi — Schlagwörter: , — BenBE @ 22:48:05

As I mentioned in an earlier post, that wasn’t the full story yet. So I’ll now keep my promise and will add the rest of the story and the missing informationen. (more…)

Flattr this!

20.11.2008

Debian Lenny and the Patches …

Filed under: GeSHi — Schlagwörter: , , , — BenBE @ 03:08:51

In May, June, July and August Milian and me were working on rewriting and optimizing some parts of the parser which caused a lot of performance gains, but also some trouble as I notice now. The problem is less that the code got broken somehow and fixed in some way again – which it sure got 😉 – but that it’s hard to reproduce how we did the fixing back then.

So what I’m refering to is a problem Secunia noted two releases after the initial patch for it. Back in GeSHi 1.0.7.22 there existed a problem, that could cause GeSHi to go into an infinite loop – I already reported about this on the news page twice. In GeSHi 1.0.8 this issue was fixed and after the release an advisory asking for an update was issued. In 1.0.8.1 work mainly concentrated on fixing the bugs in 1.0.8 thus no major work was done on the parser. The only big news in that version was a fix to a bug that had been in GeSHi since its early days and although I explicitely said there’s no real use for it in an attackers point of view Secunia started to report it which got the stone rolling.

Three days later – on the 3rd of November – I got mail from Romain Beauxis from the Debian Maintainance team caring about the php-geshi package asking me about details on the report filed by Secunia. I didn’t instantly recognize it as the bug I fixed in 1.0.8.1 but when I finally did looked up the patch I had applied back then – which was quite easy as only a few lines changed and I exactly remembered where – and sent back my answer; including a remark they should concentrate on the 1.0.7.22 bug more as that one was the more critical of both – even though Secunia didn’t even feel like mentioning it.

And that’s wher the work begins and the real trouble starts. Although I have an easy testcase there appeared a lot of questions; „What was the patch?“ being the most interesting while the least answerable of all. So the research began, only having some little hints like the approximate date of the fix, probable revisions falling in that time, the approximate place within the GeSHi source (which is about 500 lines that could be THE place!!!) and the beforementioned testcase

Following up on the mail discussion I had with the package maintainers at Debian some more details reentered the scene which shrinked the space of solutions some more. Within the changes there often have been minor changes where only few lines changed – including some changes where only formatting got changed to ease reading. All in all, about 150 revisions felt into the given range of dates, where still about 50 revisions did change geshi.php in some way – most of them affecting the faulty routine.

Besides the mentioned formatting changes there were structural and functional changes – like a new aspect of the parser getting implemented or the way some old task was performed getting optimized. If you simply skipped over the different patches there was one revision that sent out some strange „feeling“ when you read its commit comment- though you couldn’t really verify if it was the given one commit we were looking for.

So at first there had to be some look on the parts that could be affected. In parse_code there only were two loops that could easily get into hanging if you weren’t cautious enought. The first was the loop looking for the strict code blocks withing the source and the second being the loop for highlighting one such strict code block by stepping through it. And that’s where memories and luck popped in. Inspired by a mail from the Debian maintainers I where Nico Golde simplified the view on the first of the mentioned loops, I suddently remembered that Milian and me fixed some positioning code that was there to tell GeSHi where to continue after it found certain patterns. So I rechecked that portion of the code Nico had simplified in the version that was released and found the same oddity I just stumbled upon in the simplified code if Nico.

In the end I wrote a small test script and single-stepped it in the debugger … and there it was: The debugger looped exactly where I expected it to do when I was using the testcase. Given that all summed up and I verified the revision those lines were last changed: Exactly the revision I already felt strange about when I first had a broad view on that issue one week earlier.

Current state is that my findings are on their way to Nico Golde and Romain Beauxis and hopefully into an patched version 1.0.7.22-2 soon!

To sum things up on the problem you could say the following: Even if there is a patch, and you know the issue, it’s not always as simple to bring both information together as sometimes the glue for both is missing as not every issue forcibly has an ID in the bugtracking system – sometimes intentionally and sometimes because you missed to do so. What was the case with this particular issue is hard to say especially when taking into account, that you aren’t always free on your commit comments as giving to much details there might help the bad guys to be faster while being to unspecific might let you loose the connection between the comment and the problem you solved.

Remains some last topic: Why didn’t the Debian guys simply update: After some mail exchange that question was answered: They simply couldn’t as there was a feature freeze in effect prohibiting them from including any patch that added or removed any feature from the package only leaving the way open to (important) patches that focussed on correcting security issues like both the Denial of Service problem in 1.0.7.22 and the Remote Code Execution in 1.0.8 had been. And even though there had been attemps to get the newer release 1.0.8.1 into Lenny – which would have saved a lot of work – none of them was successful making up just much more work than there would have been necessary in the first place.

If you do your work later it not only gets more and harder, it also will involve much more power of others than necessary if it was done in time.

P.S.: Sorry for not being specific on some topics – I promise to get back on the details as soon as the Debian people have fixed the problem in their package.

Flattr this!

« Newer Posts

Powered by WordPress