Pickaxe - August 2015 Release Notes

by breeve 6. August 2015 13:05

See the first two tutorials (tutorial 1 and tutorial 2) to get started on the basics of the web scraper language. If you know a little SQL and something about CSS selectors, this is the tool you want. To download go here. Below is what’s new.

Proxies Statement

The language now supports an optional proxy statement which goes at the very top. With it, you can specify the proxies you want used and a test select statement. The test statement is used to test the proxies to see if they are good. If the select statement returns rows then the proxy is considered good.

The code then rotates through the proxies when running your code. Below is an example of the proxy statement. The proxies listed were free proxies from http://www.us-proxy.org/ and most likely don’t work anymore.

proxies ('', '', '')
with test {   
        pick '#tagline' take text
    from download page 'http://vtss.brockreeve.com/'

Below is a screen shot of some sample code running with the new proxy statement.

Download Image

You can now download an image into a directory like so.

insert into 'C:\Windows\temp'
from download image 'http://brockreeve.com/image.axd?picture=2015%2f6%2fheadShot.jpg'

The download image statement is a table with columns. Two of those are filename and image. The filename is a unique name generated by the runtime. The image is the actual byte[] that is saved to the file system.

Run Selected Code

Any lines of code can be selected and run much like SQL Server Management Studio. To run a segment of the code just select it and hit the run button. See below.

Stop Button

When the program is running, the stop button becomes enabled. The program can be stopped at any time. It might take a few seconds before stopping. The screenshot below shows the program in the process of stopping.

Message Window

The editor now prints out messages as the program runs in the message window. Multiple screens shots above show this.

Replace Statement

The replace statement can follow the match statement and replace all text that was matched. The below code shows us replacing the non digits with ‘reponses’ instead of ‘replies’.

    pick '.replies' take text match '\D+' replace ' reponses'
from download page 'http://vtss.brockreeve.com/'
where nodes = '.numbers'

Better Expand Statement

Now the expand statement can take an expression. The $ represents the value that changes from 1 through 10. So when the below code is run it produces rows of 11, 21, 31, 41, etc. This is perfect for generating page numbers of any repeating pattern.

select *
from expand(1 to 10) {($*10)+1}



Powered by BlogEngine.NET
Theme by Mads Kristensen

About Me

I am a Principal Engineer with 16 years experience developing and releasing software products. I started developing in C/C++ then moved into .NET and C#. Currently working in Python/Flask and Docker. Have tech lead multiple projects. I have developed products in Windows Forms, ASP.NET/MVC, Silverlight, WPF, and Python. I currently reside in Austin, Texas.

Own Projects

Pickaxe - An easy to use web page scraper. If you know a little SQL and how basic CSS selectors work, no web scraping product will be easier to use.


FligthQuery - Query FlightAware FlightXml API with SQL


Created ASP.NET MVC forum originally targeting home owner associations but now in use by an investor group.


Currently Reading