Help:External Data

From QED

The External Data MediaWiki extension with modifications has been added on a provisional basis to QED.

This extension allows formatted data from one or more sources to be included on QED pages in a fairly flexible manner. Data can come from QED pages, uploaded files or via a URL. Some filtering is supported, and tabular data can be reformatted using wikitable syntax.

Currently, the External Data extension on QED only supports XML and CSV formats, but proxy servers can be created to overcome this limitation.

The basic idea is that each data source is read using the #get_external_data function. Individual values can then be accessed using #external_value, and an entire table can be processed using #for_external_table.

Reading the data

If "URL" is a URL that returns data in one of the recognized formats, then the data can be read using the syntax suggested by the following line:

{{#get_external_data:URL|FORMAT|LOCALNAME=N|...|COL==VALUE}}

where:

• FORMAT is one of CSV, CSV with header, or XML;
• the equations LOCALNAME=N give local names to the columns, where N is the external name if there is one, or the column number otherwise;
• the forms COL==VALUE filter the data.

For example: {{#get_external_data:http://qed.princeton.edu/index.php?title=Test/test.csv&action=raw|csv|ax=1|ay=2}}

For QED pages, this can be simplified to:

• {{#get_external_data:{{fullurl:Test/test.csv|action=raw}}|csv|ax=1|ay=2}}

Here, "ax=1" means "give the name 'ax' to the first column in the dataset".

If the format is "CSV with header", then the external columns should be referenced by their external names, not by their column numbers.

The double-equality is used for filtering the data, e.g. "external_variable_name==value" selects rows satisfying the corresponding condition. Use 1 for the first column, 2 for the second, etc., if there are no external column names.

Accessing the values

To access data in a table, use the #for_external_table function. Individual values are accessed using triple-brace notation, e.g.:

{{#for_external_table:{{{ax}}} : {{{ay}}}<br> }}

Here {{{ax}}} references the column named "ax", and the text "{{{ax}}} : {{{ay}}}<br>" is repeated for each row in the table. The results are as shown in the following subsection.

A more complex example using wikitable syntax is given below.

Simple example

Combining the two lines above, we have:

These two lines together produce the following output:

Name : Color
Apple : Red
Banana : Yellow
Orange : Orange
Pear : Yellow

Tables

The following illustrates how to produce a table:

{| class=wikitable
|-
{{#for_external_table:<br/>
{{!-}}
{{!}} {{{ax}}} {{!!}} {{{ay}}}
}}
|}

Using data from a QED page or file

The official documentation for the extension currently suggests that data from a local page must be in CSV format, but that is not the case as:

1. "&action=raw" can be used on QED pages, as already illustrated above;
2. uploaded files can be accessed directly.

The following two subsections indicate how to access data on QED pages without having to spell out the full URL.

QED pages

{{fullurl:PAGENAME}} can be used to reference a specific page.

• For example, instead of giving the full URL for "Test/test.csv", we could simply write:

  {{#get_external_data:{{fullurl:Test/test.csv|action=raw}}|csv|ax=1|ay=2}}

Uploaded files

{{filepath:FILENAME}} can be used to reference an uploaded file.

• For example, instead of giving the full URL for "test.csv", we could simply write:

  {{#get_external_data:{{filepath:test.csv}}|csv|ax=1|ay=2}}

Special:GetData/PAGENAME

An alternative way to access data on QED pages is to use the Special:GetData page. This might be useful if one wishes to add meta-data to the page itself using <noinclude> ... </noinclude> tags, or if the table is large and one wishes to filter by rows.

Currently, the data must be in CSV format with the headers as the first row of data; thus, the FORMAT should be 'csv with header'.

To access the data, use a URL of the form:

• {{fullurl:Special:GetData/FULLNAMEOFPAGE}}

Or, to filter by row values:

• {{fullurl:Special:GetData/FULLNAMEOFPAGE|FIELDNAME=VALUE}}

Multiple criteria can be specified using "&", e.g. Color=Red&Shape=Round

The FIELDNAME and VALUE must be exact matches for the criteria to be satisfied.

Example

The following query:

• {{fullurl|Special:GetData/Test/test.csv|Name=Apple}}

is equivalent to:

• http://qed.princeton.edu/index.php?title=Special:GetData/Test/test.csv&Name=Apple

Please be careful about including data from a page or file to which read-access is limited. In general, such data should only be referenced on pages with the same access restrictions.

QED modifications

The External Data extension has been modified to respect access control restrictions.

Further Information

Further details may be found on the above-mentioned page.