Guides > Using data

14.30 Use your own data

Table of contents

Edit this page

This article tells you how to use your own data.

ONIX

Nielsen have kindly provided access to the ONIX 3.0 data they generate for delegates today, and we have stored that data in its raw ONIX format in your code repository. The data is in the same file format that your publishing company generates from its publishing management system (a system like our Consonance, or Klopotek or Biblio) and which it sends to data aggregators such as Nielsen, BDS, Bowker, EBSCO, JSTOR, and Amazon. By knowing how to use and reuse assets such as ONIX files to create websites and marketing materials, you get better value out of your investment in systems, and benefit from “one version of the truth”. You can populate the website you’re building today with your own ONIX 3 files, after the Day of Code, by saving your ONIX files in the _data folder.

You’ve been mainly making changes in a text editor so far, and reviewing the results in a browser. Now you’re going to use the command line to change the source of data you’re using. On the Mac, the command line is called the Terminal. On Windows, it’s called the Command Prompt. Open the relevant one now.

Troubleshoot

  • Don’t get confused between a Terminal and a text editor window. Some developers set their Terminal’s theme to have a light background and their text editor to have a dark background, or vice versa, to avoid confusion.

Your server is running in one tab, so you’ll need to add a new tab to write commands. Use the menu to do that:

Screenshot showing how to create a new terminal tab

Troubleshoot

  • Make sure you are in the correct directory. Your prompt should show you are in the “publisher-website” folder. If you are not, use the instructions from Get Started to change into the correct directory.

Transforming ONIX into JSON

To learn how to transform your ONIX data into the JSON format that your Liquid code will understand, open the README file (found at the root of your repo folder) in your text editor. It is written in Markdown: you can see it turned into HTML here.

It says:

To process the records for a publisher called snowbooks, run:

ruby lib/seed.rb --adaptor onix --publisher snowbooks

In your new Terminal tab, paste the following, changing the word “name” to be the name of the publisher you want to see the data for.

  ruby lib/seed.rb --adaptor onix --publisher name

If you are from Elsevier, you would type

  ruby lib/seed.rb --adaptor onix --publisher Elsevier

If you want to see the books from a publisher with more than one word in its name, enclose the name in quotes. For Hodder & Stoughton, you would type

  ruby lib/seed.rb --adaptor onix --publisher "Hodder & Stoughton"

If you want to see the books from all the publisher’s variations, provide just a fraction of the name. For the books from all of Penguin Adult, Penguin Books (SA) (Pty) Ltd, Penguin Books Australia, Penguin Books India Pvt Ltd, Penguin Books Ltd, Penguin Group (NZ), Penguin Putnam Inc, Penguin Random House India, and Penguin USA, you would type “Penguin”

  ruby lib/seed.rb --adaptor onix --publisher penguin

Here is a full list of all the publishers that you can choose from:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

Bloomsbury
Cambridge University Press
Institute of Physics Publishing
Librarie Philosophique J. Vrin
Morgan & Claypool Publishers
Oxford University Press
British Library Publishing
Egmont UK Ltd
Elsevier Science & Technology
Elsevier Science Publishing Co Inc
Hachette Children's Group
Hardie Grant Egmont
Hodder & Stoughton General Division
Hodder & Stoughton
Jacaranda Books Art Music Ltd
John Murray Press
Oxford University Press Inc
Oxford University Press Southern Africa
Oxford University Press, Canada
Oxford University Press
Penguin Adult
Penguin Books (SA) (Pty) Ltd
Penguin Books Australia
Penguin Books India Pvt Ltd
Penguin Books Ltd
Penguin Group (NZ)
Penguin Putnam Inc
Penguin Random House India
Penguin USA
Prelude
Profile Books Ltd
Quercus Publishing
Royal Yachting Association
SAGE Publications India Pvt Ltd
SAGE Publications Ltd
Soaring Penguin Press
Taylor & Francis

If you want to see every single book with a cover, you would type two double quotes:

  ruby lib/seed.rb --adaptor onix --publisher ""

Press enter. You’ll see a message like this:

  ONIX is being processed for Elsevier. Processed data will be put into _data/processed_books.json

Troubleshooting

  • If you see messages like lib/ adaptors/ onix/ v3/ reference/ product.rb:134: syntax error, unexpected '.' ..urrency_code}']/ PriceAmount")&.content, you are using an old version of Ruby which doesn’t support the safe navigation &.. Try rbenv global 2.5.5 or rvm use ruby-2.5.5 on the Command Line.

Now refresh your browser window and you’ll see your own data being used. Note that you didn’t change any HTML, just the source of data.

Where does this data come from?

  • As sponsors of Day of Code, Nielsen kindly allowed the Consonance team, who put together this course, access to their database of book data. The Consonance team ran searches by the publisher name of each of the delegates attending the Day of Code, to get a range of familiar products, which Nielsen created an ONIX 3.0 file for, and which the team downloaded. The code you ran to process that ONIX excludes products which do not have a cover on Nielsen, removes duplicates, and makes the publisher query case-insensitive and fuzzy (it checks to see if the publisher name contains, rather than is equal to, the query). The code is in lib/adaptors/onix/v3/reference.rb, in the products method.
  • Additionally, for the Consonance clients at Day of Code, we have used ONIX 3 data straight out of Consonance.

VERY IMPORTANT: take a screenshot!

On Monday 25th November, your hosts from Consonance have to present a summary of the Day of Code at the FutureBook conference. We need screenshots from every delegate to showcase our results! Please screenshot your portfolio page, and index page, and send the images to us now via Slack or email: info@consonance.app. Thanks!

Learn more later

  • You have used a library written by the Consonance team to transform ONIX 3 into JSON which Jekyll can understand. Go to lib/write_data.rb to see it, and run the tests by typing rspec on the command line. And Google the words I’ve just used: library, JSON, rspec.

What you’ve learned

  • You can reuse data sources to get the most out of your investment in your systems.
  • Using one data source means you have one version of the truth, which is more efficient.