Category: Uncategorized

Ever wondered how scraping works? If you are pretty much blank when it comes to programming, this guide is probably not for you. However, if you have the basic concepts in place, in a few steps the author, Mikko Helsig, shows you how to scrape a site in Python (and also how to install Python in a Windows environment).

Prerequisites are a basic understanding of programming. But then you get a concept of how Python works, how scraping works and the really cool libraries requests, requests_cache, BeautifulSoup and Gender (the latter is a library used to guessing and parsing gender of names).

Link: Scraping 101

If you are totally new to programming, we encourage you to start by learning a little Python. There are numerous places to do this. For instance:

• Codecademy (both free and $)
• Learn Python (free)
• W3Schools Python Tutorial (free)

We also deeply encourage you to start with programming in an environment such as Anaconda. A short description of the Anaconda Navigator can be found here.

We have added Instaloader to our External resources.

Instaloader is a tool to download pictures (or videos) along with their captions and other metadata from Instagram. You can either download profiles or hashtags, and it’s possible to set up filters (for instance datefilters, see below) to narrow your search.

To use Instaloader,  you should do the following.

1. Download and set up a new Anaconda Environment with a Python version higher than 3.5.
2. Install Jyputer Notebook on the environment and open a new terminal
3. Do a pip (not pip3, as that does not work with Anaconda) install of the instaloader and dependencies

   pip install instaloader

4. Create a new folder in your root-environment (typically documents-folder) called for instance Instaloader
5. In terminal do

   cd instaloader

6. This is to avoid that everything is saved in your base folder 🙂
7. Run various command line commands in your terminal. Please do note that the interface is rudimentary but filters can be applied with the use of boolean expressions for instance:

   instaloader "#HASHTAG" --post-filter="date_utc >= datetime(2017,1,1) and date_utc <= datetime(2018,1,1)" --login=USERNAME

8. We would love to implement this as a hosted service. However, it is not likely we will do so just now. Therefore, please experiment with it yourself. You can also ask our advice, and we will do our best to help. If you plan to use this tool on a regular basis or for larger datasets, you should probably be ready to use several user accounts and/or proxies to avoid being banned.

On 28/11 2018 we had our official launch with a seminar and get-together at our place.

We had a smashing time discussing the possibilities with you guys, and look forward to realizing ideas from the day. If you would like to relive the day (and who wouldn’t), you will find links to presentations and programme below.

• Opening presentation by Jannie Møller Hartley
• Presentation by Sander Schwartz
• Presentation by Mads Kæmsgaard Eberholst
• Presentation by Eva Mayerhöffer
• Launch Programme

Our hosted service for collecting Twitter data called Yourtwapperkeeper is currently retired and is having a lovely time at the old software home.

Currently the data can still be downloaded, but the server will be completely closed in January 2019. If you are using data from that server, you should download it by then.

If you need Twitter-data collection please use our TCAT-servers instead.

All data and ongoing collections from Yourtwapperkeeper has already been migrated to TCAT. Read more about the TCAT services on our Hosted resources.

I det mange tilfælde oplever vi, at studerende og forskere helst vil tage kontakt med os via e-mail, men i efterårssemesteret 2018 eksperimenterer vi med begrænset åbningstid.

Indenfor den begrænsede åbningstid vil Sander sidde på kontoret. Du kan således benytte lejligheden til at komme uanmeldt, men ofte kan det være en god idé at skrive inden, så Sander ved, at du kommer og eventuelt kan forberede sig på at løse dit problem.

Åbningstiderne er i øjeblikket hveranden onsdag i perioden 13-15. Det drejer sig således om følgende datoer i 2018:

• 24/10, kl. 13-15
• 07/11, kl. 13-15
• 21/11, kl. 13-15
• 05/12, kl. 13-15
• 19/12, kl. 13-15

We are alive and kicking!

We are starting up our toolbase with some rudimentary tools and scripts. They are divided by Hosted resources and External resources.

Hosted resources are things we are hosting ourselves on local servers. Or scripts, that we have made ourselves and can assist you with running on your own computer or in the lab by appointment.

External resources are things we think are interesting, but really have no ownership over. Tools you can use, but we don’t officially support. You are free to ask advice on the tools too, though.

We also have a list of links on where you can find data. You could browse that list, but it is far far far from complete. So if you know a good link, please send us an e-mail.