Finding and downloading images within the Wikipedia Dump -


i'm trying find comprehensive list of images on wikipedia, can filter down public domain ones. i've downloaded sql dumps here:

http://dumps.wikimedia.org/enwiki/latest/

and studied db schema:

http://upload.wikimedia.org/wikipedia/commons/thumb/4/42/mediawiki_1.20_%2844edaa2%29_database_schema.svg/2193px-mediawiki_1.20_%2844edaa2%29_database_schema.svg.png

i think understand when pick sample image wikipedia page can't find anywhere in dumps. example:

http://en.wikipedia.org/wiki/file:carrizo_2a.jpg

i've done grep on dumps 'image', 'imagelinks', , 'page' looking 'carrizo_2a.jpg' , it's not found.

are these dumps not complete? misunderstanding structure? there better way this?

also, jump ahead 1 step: after have filtered list down , want download bulk set of images (thousands) saw mentions need mirror of site prevent overloading wikipedia/wikimedia. if has guidance on too, helpful.

mediawiki stores file data in 2 or 3 places, depending on how count:

  • the actual metadata current file versions stored in image table. want; you'll find latest en.wikipedia dump of here.

  • data old superseded file revisions moved oldimage table, has same structure image table. table dumped, latest 1 here.

  • finally, each file (normally) corresponds pretty ordinary wiki page in namespace 6 (file:). you'll find text of these in xml dumps, same other pages.

oh, , reason you're not finding files linked in english wikipedia dumps they're shared repository @ wikimedia commons. you'll find them in commons data dumps instead.

as downloading actual files, here's (apparently) official documentation. far can tell, mean "bulk download (as of september 2012) available mirrors not offered directly wikimedia servers." if want images in tarball, you'll have use mirror. if you're pulling relatively small subset of millions on images on wikipedia and/or commons, should fine use wikimedia servers directly.

just remember exercise basic courtesy: send user-agent string identifying , don't hit servers hard. in particular, i'd recommend running downloads sequentially, start downloading next file after you've finished previous one. not easier implement parallel downloading anyway, ensures don't hog more share of bandwidth , allows download speed more or less automatically adapt server load.

ps. whether download files mirror or directly wikimedia servers, going need figure out directory they're in. typical wikipedia file urls this:

http://upload.wikimedia.org/wikipedia/en/a/ab/file_name.jpg

where "wikipedia/en" part identifies wikimedia project , language (for historical reasons, commons listed "wikipedia/commons") , the "a/ab" part given first 2 hex digits of md5 hash of filename in utf-8 (as they're encoded in database dumps).


Comments

Post a Comment

Popular posts from this blog

monitor web browser programmatically in Android? -

i've got tricky question here. need users make payment bank (namely barclaycard) in uk. so, have https url , add parameters (such amount pay, order reference, etc) url, start http connection intent.actionview, redirect user browser can enter credit card details on bank's webpage , make payment our account successfully. far ? the code use below (i changed values privacy reasons) problem is, need app when user has completed/failed/cancelled payment. barclaycardautomatically redirects particular url when payment has succeeded, 1 if failed. there no way of knowing when barclaycard payment has succeeded go android app somehow ? button cardbutton = (button) findviewbyid(r.id.card_button); cardbutton.setonclicklistener(new view.onclicklistener() { @override public void onclick(view arg0) { string prehashstring = new string(); string prohashstring = new string(); string shapassphrase = new string(); shapassphrase = "gsvth£h70zkh...
Read more

Shrink a YouTube video to responsive width -

i have youtube video embedded on our website , when shrink screen tablet or phone sizes stops shrinking @ around 560px in width. standard youtube videos or there can add code make go smaller? you can make youtube videos responsive css. wrap iframe in div class of "videowrapper" , apply following styles: .videowrapper { float: none; clear: both; width: 100%; position: relative; padding-bottom: 56.25%; padding-top: 25px; height: 0; } .videowrapper iframe { position: absolute; top: 0; left: 0; width: 100%; height: 100%; } the .videowrapper div should inside responsive element. padding on .videowrapper necessary keep video collapsing. may have tweak numbers depending upon layout.
Read more

wpf - PdfWriter.GetInstance throws System.NullReferenceException -

i'm trying create pdf document using itextsharp 5.3.4 using following document document = new document(); filestream stm = new filestream(filename, filemode.create); pdfwriter writer = pdfwriter.getinstance(document, stm); i'm getting system.nullreferenceexception following stack trace: system.nullreferenceexception occurred hresult=-2147467261 message=la référence d'objet n'est pas définie à une instance d'un objet. source=itextsharp stacktrace: à itextsharp.text.version.getinstance() innerexception: i've verified neither document nor stm null, , if select "continue" in vs12 document created - exception thrown. updated itextsharp 5.4.0 , it's still occurring. can't find information on anywhere - got ideas? make sure not catching exceptions. nullreferenceexception can 1 caught , handled inside itextsharp, don't care it. fact can continue supports theory. change following setting verify: debug -> exceptions ...
Read more