Contact Us

The main tasks of the gwebcmd utility are loading web data and converting web data and text files into CSV format.

gwebcmd allows loading web data using http and https, GET and POST web requests or using Internet Explorer.

gwebcmd can parse the following web data formats:

  • XML
  • JSON
  • HMTL
  • CSV
  • Plain text

gwebcmd allows getting protected web data and supports the following authorization methods:

  1. Windows
  2. Basic
  3. Forms
  4. OAuth 1.0
  5. OAuth 2.0

The data in CSV format are commonly used for further import into databases.

You may use the companion gsqlcmd command line utility to import CSV data into databases.

If you need to use the data in Microsoft Excel, you may use SaveToDB add-in for Microsoft Excel that allows loading web data into Microsoft Excel.

Getting and Converting Web Data into CSV

The webtocsv mode is used to get the data and save it to a csv file.

The command line format is

gwebcmd webtocsv  <URL> [<output file name>] [<options>]

Examples of getting data from XML data sources:

gwebcmd webtocsv https://api.datamarket.azure.com/WorldBank/WorldDevelopmentIndicators/v1/GetCountries?LanguageCode=%27en%27 GetCountries.csv
gwebcmd webtocsv http://api.worldbank.org/incomeLevels/LIC/countries countries.csv
gwebcmd webtocsv https://api.linkedin.com/v1/people/~ linkedin_info.csv
gwebcmd webtocsv https://docs.google.com/feeds/default/private/full?v=3 google_docs.csv
gwebcmd webtocsv http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.quotes%20where%20symbol%20in%20%28%22AAPL,GOOG,YHOO%22%29&diagnostics=false&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys quotes.csv
gwebcmd webtocsv http://social.yahooapis.com/v1/user/me/contacts yahoo_contacts.csv

Examples of getting data from JSON data sources: 

gwebcmd webtocsv https://apis.live.net/v5.0/me/contacts live_contacts.csv
gwebcmd webtocsv https://graph.facebook.com/me/friends facebook_friends.csv
gwebcmd webtocsv http://www.google.com/m8/feeds/contacts/default/base?alt=json google_contacts.csv
gwebcmd webtocsv http://api.linkedin.com/v1/people/~/connections:(id,first-name,last-name)?format=json linkedin_connections.csv
gwebcmd webtocsv https://api.twitter.com/1.1/statuses/user_timeline.json twitter_user_timeline.csv

Examples of getting data from CSV web pages:

gwebcmd webtocsv http://www.google.com/finance/historical?q=AAPL&output=csv google_csv_aapl.csv
gwebcmd webtocsv http://ichart.finance.yahoo.com/table.csv?s=AAPL&ignore=.csv yahoo_csv_aapl.csv

Examples of getting data from HTML web pages:

gwebcmd webtocsv http://finance.yahoo.com/q/op?s=AAPL yahoo_options_aapl.csv
gwebcmd webtocsv http://finance.yahoo.com/q/hp?s=AAPL+Historical+Prices yahoo_htm_aapl.csv
gwebcmd webtocsv http://www.google.com/finance/historical?q=AAPL google_htm_aapl.csv 

Getting Web Data using Internet Explorer and Converting into CSV

Loading web data using Internet Explorer is either the same as loading using web requests.

The basic command line is

gwebcmd ietocsv <URL> [<output file name>]

gwebcmd starts Internet Explorer, performs specified in the /actions option actions, saves the page, and closes Internet Explorer.

This option is useful for loading data from protected pages or from pages updated using AJAX.

See the IE Options topic to learn available options.

Examples:

gwebcmd ietocsv "http://finance.yahoo.com/q/hp?s=AAPL+Historical+Prices" aapl.htm
gwebcmd ietocsv "http://www.google.com/" "excel mysql.csv" "/actions=gbqfq=excel mysql;{enter}" /googleWebSearch

IeToCSV is a premium mode of the Personal and Enterprise Editions.
gwebcmd Express Edition has a delay in 30 seconds.

Customizing CSV Data

The following options are used to customize output CSV data:

/datetimeformat=<format>
/separator=<separator>|Tab
/add=<header=value>[<separator>...]

You may define required format for datetime values and a column separator, and also add constant data to a CSV output.

For example:

gwebcmd webtocsv http://finance.yahoo.com/q/hp?s=AAPL+Historical+Prices yahoo_htm_aapl.csv /datetimeformat=yyyy-MM-dd /separator=; /add=Symb=AAPL

The first rows of the result file:

_RowNum;Symb;Date;Open;High;Low;Close;Volume;Adj Close*
0;"AAPL";2014-07-07;420.86;422.98;417.45;420.8;8590100;420.8
1;"AAPL";2013-07-02;409.96;421.63;409.47;418.49;16780900;418.49
2;"AAPL";2013-07-01;402.69;412.27;401.22;409.22;13966200;409.22
3;"AAPL";2013-06-28;391.36;400.27;388.87;396.53;20661300;396.53

Getting Data from Multiple URLs

Multiple requests to websites should be executed with a delay between requests to prevent banning your IP by web servers.

Websites specify this requirement in the Terms of Service document.

Use the sleep mode or the /delay option to specify a delay in command files, in milliseconds.

For example:

gwebcmd sleep 500

Here is an example of a simple batch file to get the data for a list of tickers:

@echo off
set gwebcmd="C:\Program Files (x86)\Gartle\gwebcmd\gwebcmd.exe"
@for /F %%i in (tickers.txt) do (
echo %%i
%gwebcmd% webtocsv http://finance.yahoo.com/q/hp?s=%%i+Historical+Prices %%i.csv /datetimeformat=yyyy-MM-dd /add=Symb=%%i
%gwebcmd% sleep 500
)

The gwebcmd variable is used to specify the full path of the gwebcmd executable.

You may add the path to the PATH variable and use gwebcmd.exe instead.

The tickers.txt file contains a ticker per line. For example:

AAPL
FB
GOOG
LNKD
MSFT
YHOO

You may also use task files to get data from multiple pages.

For example, a task file, task.txt, can contain:

 CsvFileName  URL
csv\aapl.csv http://ichart.finance.yahoo.com/table.csv?s=AAPL&ignore=.csv
csv\goog.csv http://ichart.finance.yahoo.com/table.csv?s=GOOG&ignore=.csv

and the batch file can be much simpler:

@echo off
set gwebcmd="C:\Program Files (x86)\Gartle\gwebcmd\gwebcmd.exe"
%gwebcmd% webtocsv @task.txt /datetimeformat=yyyy-MM-dd /delay=500

Getting Web Data Using Web Requests

The webtotext mode is used to get the text of a web page and to save it to a file.

The basic command line format is

gwebcmd webtotext <URL> [<output file name>]

For example:

gwebcmd webtotext http://finance.yahoo.com/q/hp?s=AAPL+Historical+Prices aapl.htm

Getting Web Data Using Internet Explorer

The ietotext mode is used to get the text of a web page and to save it to a file.

Using Internet Explorer allows loading data from protected pages or from pages updated using AJAX.

See the IE Options topic to learn available options.

The basic command line format is

gwebcmd ietotext <URL> [<output file name>]

For example:

gwebcmd ietotext http://finance.yahoo.com/q/hp?s=AAPL+Historical+Prices aapl.htm

IeToText is a premium mode of the Personal and Enterprise Editions.
gwebcmd Express Edition has a delay in 30 seconds.

Parsing Text to CSV

The texttocsv mode is used to parse the text of a saved web page and save the CSV data to a file.

The command line format is

gwebcmd texttocsv <input file name> [<output file name>] [<Options>]

For example:

gwebcmd texttocsv aapl.htm aapl.csv

The options allow customizing CSV output and are the same as described above.

Getting Data from Multiple Pages

You may load and parse web data specifying URLs for each required page.

Another way is to specify the /pages=<number of pages> option.
gwebcmd will try to find next page URLs and to load multiple pages.
This way works if gwebcmd successfully finds the next page URLs.

This is premium feature of the Personal and Enterprise Editions.
gwebcmd Express Edition allows loading two pages.

Getting Option Chains from Yahoo! Finance and Google Finance

You may load and parse option chain pages from Yahoo! Finance and Google Finance.

For example, you may load AAPL options with the expiration date on January 17, 2015:

gwebcmd webtocsv "http://finance.yahoo.com/q/op?s=AAPL&date=1421452800" aapl-150117.csv
gwebcmd webtocsv "http://www.google.com/finance/option_chain?q=AAPL&authuser=0&output=json&expy=2015&expm=1&expd=17" aapl-150117.csv /rootPath=calls,puts

The challenge is to get option chains for all expiration dates.

The simplest way is to specify:

gwebcmd webtocsv "http://finance.yahoo.com/q/op?s=AAPL" aapl.htm aapl.csv /pages=20
gwebcmd webtocsv "http://www.google.com/finance/option_chain?q=AAPL&authuser=0&output=json" aapl.json aapl.csv /rootPath=calls,puts /pages=20

gwebcmd will load up to 20 pages of options chains starting from the first page (without the date in the example).

This is a premium feature.
gwebcmd Express Edition allows loading 2 pages only, Personal Edition - option chains with expiration in the nearest 3 months, and Enterprise Edition allows loading all option chains.

Getting Google Web Search Results

You may load and parse Google web search pages.

For example, you may load search results for the "excel mysql" keyword using the following commands:

gwebcmd webtocsv "https://www.google.com/search?as_q=excel%20mysql&num=20"          "excel mysql-01.csv" /googleWebSearch
gwebcmd webtocsv "https://www.google.com/search?as_q=excel%20mysql&start=20&num=20" "excel mysql-02.csv" /googleWebSearch

Note that you have to specify the /googleWebSearch option to parse the page as Google web search results.

Using the /page option you may load results with fewer efforts:

gwebcmd webtocsv "https://www.google.com/search?as_q=excel%20mysql" "excel mysql.csv" /googleWebSearch /pages=5

This is a premium feature.
gwebcmd Express Edition allows loading 2 pages only, Personal Edition - 5 pages, and Enterprise Edition allows loading unlimited number of pages.

To Top