2013-11-28

Querydsl user survey results

A few weeks ago we opened the Querydsl user survey to gather some information on how Querydsl is used and what direction it would make sense to take it. Some of the results were a little surprising, but mostly the results confirmed that the decisions we had made in the past were mostly right and that most of ours users are extremely happy with the product.

Here is a summary of the results, the amounts are in percentages:

How have you heard of Querydsl?

Which IDEs are you using?

Which build tools are you using with Querydsl?

Which Querydsl modules are you using?

What other databases would you like to see supported in Querydsl?

What JDK version/s are you using?

What is your overall satisfaction with Querydsl?

What are the major benefits in using Querydsl?

How have you participated in the Querydsl community?

What commercial Querydsl services would you be interested to try?

Do you have anything else to tell us

This question was not multiple-choice, so only the most common topics are listed below

  • Better docs, more examples
  • Roadmap of coming features
  • Donation page
  • Better readability for Querydsl SQL queries
  • Dedicated Gradle support
  • More advanced SQL functionality
  • Continue with the current licensing model
  • Better control of code generation target packages
  • Better docs for Hibernate Search and Collections modules

Conclusion

Based on these promising results we will take the following actions:

  • Expand our commercial offers
  • Invest into Querydsl SQL, since it is the second most popular Querydsl module
  • Improve the general documentation
  • Provide a public Roadmap for Querydsl development

Feedback on Querydsl and its development model is of course also welcome in any other form. Let us know what features you are missing and what use cases should be supported better.

2013-11-21

Hibernate challenges: Entity persistence

Hibernate has been the most used persistence technology in our Java projects. We have optimized Hibernate usage via our own tooling such as Querydsl JPA for querying as well as best practices and conventions for other parts such as entity lifecycle handling.

In this blog post I’d like to address Hibernate issues related to entity persistence and loading. Some of these challenges we have faced ourselves in customer projects and others are issues our customers have come across.

Usage of right session level methods

Hibernate provides multiple methods for persisting entities with slightly different semantics. To understand the methods better it is good to understand the different states Hibernate managed entities go through.

The starting point for new entities is the transient state. Transient entities are unknown to any Hibernate session and don’t have any database identity attached. Transient instances can be persisted by Session.save() and Session.saveOrUpdate(). save will assign the database identity value to the entity’s id property, but might not do any SQL INSERTs. Session.save() should be seen as the way to register an entity with Hibernate.

The persistent state is the state loaded and queried entities are in. Also transient entities become persistent when saved via Session.save() or saveOrUpdate(). Persistent instances are tracked by the Hibernate session and changes are persisted to the database at session flush time.

Persistent entities become detached via Session.evict(), Session.clear() and Session.close() calls. Detached entities do have a database identity, but are not attached to a running entity. Detached entities are usually used when entities are used in the view rendering phase for web application with an active session or in long running conversions with multiple sessions.

Persistent entities can be also be deleted via Session.delete(). The deletions on the database level happen at session flush time.

There is also a path back from detached entities to persistent entities via Session.saveOrUpdate(), Session.update() and Session.merge().

A common problem with Hibernate session usage is that methods are not used in the right way or unnecessary session level method calls are made. The following two cases are quite common examples of wrong API usage.

It is quite common to see method calls to persist entities which are known to be managed by Hibernate.

Person person = new Person();
session.save(person);
// modifications to person
session.update(person);

In this case the update call in the end is a NOOP since the person instance is already persistent.

Another common antipattern is to control the session size with too fine grained calls.

Person person = new Person();
// modifications to person
session.save(person);
session.flush();
session.clear(); // or session.evict(person)

The code above makes the person instance persistent, assigns and id, saves it to the database on flush and clears the session on clear.
There are several problems with this approach

  • JDBC batching on INSERT and UPDATE can’t be utilized, since flushes happen too often
  • other entities are also removed from the session which might break surrounding code, evict helps in this case
  • the person instance is detached after this call and that needs to be taken into account in it’s usage

Explicit flush and evict calls are often necessary for batch operations, but for other cases implicit flushes should be used on transaction ends.

Mixing session level and SQL level manipulations

Hibernate provides the possibility to mix session level operations on entities with direct updates to the database via DML clauses. When these get mixed things get complicated

List<MCustomer> customers = session.createQuery("select c from Customer c").list();

String hql = "update Customer c set c.name = :newName where c.name = :oldName";
int updatedEntities = s.createQuery(hql)
        .setString("newName", newName)
        .setString("oldName", oldName)
        .executeUpdate();

for (Customer customer : customers) {
    // modifications to customers
}

In this case the modifications to the customers on the Java level override the modifications via the DML clause, even if other properties are modified. Also the updates executed via the update are not reflected in the second level cache.

A good solution to this problem is to keep DML and session level operation usage in different transactions. If that is not possible one should be very careful not to create inconsistent end results.

Efficient batch inserts and updates

The promoted approach to batch insert entities via session level methods is

for (int i = 0; i < 100000; i++) {
    Customer customer = new Customer(.....);
    session.save(customer);
    if (i % 20 == 0) { //20, same as the JDBC batch size
        //flush a batch of inserts and release memory:
        session.flush();
        session.clear();
    }
}

This is a good approach if used in an isolated session. If other things happen before and after this code in the same session a more secure approach could be

Set<Customer> batch = new HashSet<Customer>();
for (int i = 0; i < 100000; i++) {
    Customer customer = new Customer(.....);
    session.save(customer);
    batch.add(customer);    
    if (batch.size() == 20) { //20, same as the JDBC batch size
        for (Customer c : batch) {
            session.evict(c);
        }
        session.flush();
        batch.clear();
    }
}

The same applies for batch updates. Usage of session.clear() in a more complex context is always risky.

Efficient loading

Efficient loading of entities and collections involves tuning the loading behaviour when querying.

It is usually a good practice to favor query scope optimizations of loading and fetching instead using annotations extensively, because this gives you more flexibility and emphasizes use case specific optimizations instead of general behaviour.

Collections are by default loaded lazily but they can fetched in a single select via the fetch keyword

select c from Company c
left join fetch c.departments

As an annotation inside Company it could be declared like this

    @Entity
    public class Company {

        @Fetch(FetchMode.Join)
        private Set<department> departments;
    }

It is usually good to define as annotations the desired behaviour for loading via session.get() or session.load() and via query flags the optimizations for loading of multiple entities via a query.

Hibernate or JPA API?

Usage of the JPA 2 API is usually promoted in favor of Hibernate’s own Session API. Even the Hibernate team itself pushes the adoption of the JPA API http://www.theserverside.com/news/2240186700/The-JPA-20-EntityManager-vs-the-Hibernate-Session-Which-one-to-use.

In one of our projects we migrated from Hibernate Session usage to JPA API and the biggest pain points were related to the methods related to persistence of entities and related cascading operations. Queries were easily ported since they were already handled by Querydsl which supports both APIs.

The main methods in Hibernate to persist objects are session.save() and session.update(). In JPA they are entityManager.persist() and entityManager.merge(). Hibernate provides additionally the saveOrUpdate convenience method, which will use either save or update depending on the state of the object. A saveOrUpdate() equivalent is not available in the JPA and the entityManager.merge() method is not a direct replacement for session.update(), so it is not a trivial job to replace saveOrUpdate() usage when migrating to the JPA API.

The challenges we faced were

  • no saveOrUpdate() replacement available and own implementation proved to be not fully compatible with the previous behaviour
  • Hibernate cascade annotations didn’t work reliably anymore when EntityManager persistence methods were used.

We ended up replacing the Hibernate cascade annotations with JPA cascades and replacing saveOrUpdate calls with a custom persistOrMerge method which used the JPA API.

Having a strong DAO / Service division helps with migrating from one persistence API to another, but since the transaction boundary is often on the Service layer the persistence API behaviour often leaks to the service layer.

For a more indepth discussion of this problem read this blog post http://blog.xebia.com/2009/03/23/jpa-implementation-patterns-saving-detached-entities/

2013-10-31

Querydsl commercial offers

We are frequently asked about our future plans regarding Querydsl and in which direction we intend to take it commercially. With the recent announcements of the jOOQ project to change its licensing model from a pure OSS licensing model to dual licensing our users are wondering if we might follow suit.

We have no intention of changing the licensing model of Querydsl and will keep on using a pure open source license for the foreseeable future. We feel that middleware should be transparent, and open source is one of the best models to guarantee both transparency and an interactive development model.

Our commercial model for Querydsl is open source software and commercial services. Querydsl is free to use and you will get free public support for it, but for private support, bigger additions and closed source extensions we are available for consulting and development at reasonable hourly rates.

Training

Querydsl training is available in packages and can also be customized for special needs. The available packages are

Introduction

one day of training to get an overview of the general functionality of Querydsl and usage examples for its most popular modules JPA and SQL

Querydsl & JPA/Hibernate

two days of training to get an overview of Querydsl and how it is best used with JPA including best practices for service/DAO design with Querydsl

Querydsl & SQL

two days of training to get a general overview of Querydsl and how it is best used with JDBC/SQL

Project development

Development services for projects involving Querydsl and related technologies are also available. We have worked with small startups and big global corporations to provide solutions using open source Java and Web technologies.

See our references for some examples of projects we’ve done in the past.

In addition, we have consulted a major Finnish software company on Hibernate performance problems. And later given training on Hibernate and Querydsl usage for their development team in St. Petersburg.

Extensions

Querydsl extensions in both open source and closed source form can be provided by us via development projects.

Examples of such extensions are
  • support for SQL/NoSQL technologies that are not yet supported by Querydsl (e.g. DB2 or Cassandra)
  • higher level abstractions for Querydsl that fit into your IT infrastructure
  • integration of Querydsl into other middleware frameworks or libraries

Contact us at querydsl@mysema.com if any of these options sound attractive to you. And if you haven’t yet participated in the Querydsl user survey, please do so.

2013-10-07

HelMet integration for Linked Events

The most recent addition to Linked Events, the event information aggregator developed by Mysema, is the integration of events taking place at libraries throughout the Helsinki region. This information is kindly provided by HelMet.

Adding new sources of data to an existing system is always a challenge. For example, the format of the new data might be incompatible with the existing data. This is true for event information in particular since there isn’t any generally accepted format, although some work is being done in that direction, see for example schema.org, EventsML or CitySDK. Other difficulties can arise from incompatibilities at the API level - even though a machine-friendly API is provided, it isn’t necessarily cut out for all possible use cases that application developers might come up with.

Open data has become a priority for the public sector in many countries. The idea is enticing - make your data open and let people innovate with it, developing applications that the public sector itself does not have the resources to implement. New York has NYC BigApps, Helsinki loves developers and there is even an Apps4Pirkanmaa competition, to name just a few. The HelMet case is a good example of a public organization’s willingness to expose data for public use - information about events taking place in HelMet’s network of libraries is provided using a machine-friendly API, which we decided to leverage in the Linked Events portal.

The challenges of open data

Despite the easy-to-use API, we still encountered a few problems while integrating our new data source, problems that probably are rather typical for developers trying to leverage open data. In our case, several vital pieces of information were missing from the event data that the API provides - location information, translations and so on. All of these issues would certainly be possible to fix, but it would probably take a long time for the data provider, a large public sector organization, to go through the hoops of procurement, IT development outsourcing and so forth. Because of time constraints, we ended up scraping a website for the missing pieces of information, an unnecessarily brittle step in what could otherwise have been a very simple integration.

Developers of applications that utilize open data are usually individuals or small, agile companies. When the providers of the open data are large organizations that cannot respond quickly to developers’ needs, there is going to be friction. Here are some tips for organizations that want developers to be able to smoothly leverage their open data sets:

  • Prefer open source components and universal data formats for developing APIs. This avoids provider lock-in and makes maintenance easier.
  • Outsource carefully and have someone in your organization with sufficient technical know-how oversee the project. Facilitate knowledge transfer by bringing consultants in-house and having them collaborate with your own people instead of doing development off-site.
  • Designate someone to be the “API maintainer”. This person should coordinate development, answer questions from developers, monitor API usage and watch out for potential problems. Provide this person’s contact information to developers. 
  • Open a public issue queue on for example GitHub. This provides a channel for developers to report issues, to discover what problems other developers have run into and and to see what progress has been made on resolving them.
  • Transparency is key - provide documentation, examples, a list of known bugs and so on. Consider open-sourcing your API. 

I think the push for making data open and accessible is an applaudable effort by the public sector. There have already been many beautiful applications built using open data - take, for example, BlindSquare or our own Linked Events aggregator. To make open data even more useful, data providers should seek to become more agile and responsive to developers’ needs. A public-facing API can’t be implemented and then forgotten about, it is something that needs to be supported and refined continuously.

2013-09-25

Querydsl in Finnish eDemoracy and eServices projects

Querydsl, our open source database query library, is used in multiple business sectors to make querying databases easier and more productive. Besides various private sector projects Querydsl is used at least in two large-scale public sector projects of the Action Programme on eServices and eDemocracy (SADe) of the Finnish Ministry of Finance here in Finland.

The main goal of the SADe project is to develop public services for individuals, companies and other entities. These customer-oriented and interoperable services are meant to increase the quality and efficiency of public sector services. The SADe programme consists of eight projects which are managed by the Ministry of Finance and other ministries:

Querydsl is used in at least two of these projects (source code will be open by the end of this year):

In the Finnish eParticipation Environment of the Ministry of Justice, or more specifically in the Kansalaisaloite.fi portal, Querydsl is used on top of SQL databases. We have found that handwritten SQL is usually hard to maintain, error-prone, and slow. Querydsl basically solves all of these problems.

Learners’ Online Services of the Ministry of Education and Culture is also using Querydsl. Their use case is a bit different: they have SQL and NoSQL databases, Querydsl keeps the developers productive when switching between the two due to its consistent query API. Mysema has provided consulting to the Learners’ Online Services developers on how to get the most out of Querydsl.

We used Querydsl on top of Apache Lucene when developing Suomen Museot Online. This site provides a browsable database of almost 200,000 museum items. It gathers data from nearly 50 Finnish museums and makes the data available to Europeana.

If you are interested in Mysema’s consulting services, please contact sales@mysema.com.

2013-09-03

Euroopan unionin tietosuojalainsäädäntö uudistumassa

(This post is only available in Finnish)

Kiitos Edward Snowdenin tietoturvasta ja henkilötietojen suojaamisesta on tullut viime aikoina ajankohtaisia puheenaiheita. Yhdysvaltojen PRISM-ohjelman laajuus on tietenkin tullut yllätyksenä, mutta ohjelmistojen suunnitteluun ja implementointiin meillä Suomessa se ei kuitenkaan suoraan vaikuta. Toinen, paljon merkittävämpi seikka, joka ei ainakaan tähän asti  ole hirveästi julkisuutta saanut, on Euroopan unionin uusi tietosuojalainsäädäntö.

Uutta tietosuoja-asetusta säädetään tällä hetkellä Euroopan unionissa ja se tulee voimaan mahdollisesti noin kahden vuoden päästä. Asetuksena se tulee jäsenvaltioissa tietyn siirtymäajan jälkeen automaattisesti voimaan, toisin kuin esimerkiksi direktiivi, jota jäsenvaltio pystyy vielä muokkaamaan ennen voimaanastumista. Suomen kohdalla uusi asetus korvaa henkilötietolain vuodelta 1999.

Keskustelu uudesta tietosuojalainsäädännöstä on tähän asti kohdistunut isoihin yrityksiin, esimerkiksi Facebookiin ja Googleen, jotka prosessoivat valtavia määriä henkilötietoja. Pitää kuitenkin muistaa, että uusi lainsäädäntö tulee koskemaan kaikkia tahoja, yrityksiä ja organisaatioita, jotka jollain tavalla käsittelevät henkilötietoja. Yhtä paljon lainsäädäntö koskee Suomen valtiota, kuntia, seurakuntia ja vaikka videovuokraamoja kuin kansainvälisiä Internet-yrityksiä. Uuden asetuksen myötä laiminlyönti tietosuoja-asioissa voi johtaa isoihin sakkoihin.

Uuden asetuksen noudattaminen vaatii, että henkilötietojen suojaaminen ja oikeaoppinen käyttö otetaan alusta asti huomioon ohjelmistoprojektien suunnittelussa ja toteutuksessa. Tietosuoja ei tietenkään ole ohjelmistokehitysprojekteissa mikään uusi vaatimus, mutta tekniset haasteet lisääntyvät tuntuvasti uuden asetuksen myötä. On esimerkiksi ehdotettu, että henkilöllä, jonka tietoja järjestelmässä käytetään, tulisi olla mahdollisuus kaikkien omien tietojen poistamiseen. Tietoja pitää myös pystyä viemään ulos järjestelmästä ”yleisesti käytetyssä formaatissa” helpottaakseen siirtoa yhdeltä palveluntarjoajalta toiselle. Facebookin kokoisella yrityksellä on tarpeeksi resursseja toteuttaa järjestelmissään tällaiset ominaisuudet. Pienen kunnan rakennuslautakunnalle tai vaikka musiikkikoululle uudet vaatimukset voivat kuitenkin olla hyvinkin haasteellisia.

Uuden asetuksen myötä henkilötietoja käsittelevälle organisaatiolle syntyy uusia velvollisuuksia. Mahdollisista tietosuojaloukkauksista tulee esimerkiksi ilmoittaa tietosuojaviranomaiselle ja käyttäjille. Organisaation pitää pystyä näyttämään, että henkilötietoja ei tallenneta vailla käyttötarkoitusta, ja että ne poistetaan automaattisesti, kun niitä ei enää tarvita.

On todennäköistä, että uuden tietosuoja-asetuksen vaatimukset - etenkin tekniset - voivat tulla yllätyksenä monelle tietojärjestelmän tilaajalle. Mysemalla seuraamme tietosuojalainsäädännön kehitystä tarkasti ja otamme tarvittaessa asian hyvissä ajoin puheeksi, jotta asiakas voi olla varma olemassa olevan lainsäädännön noudattamisesta.

Euroopan unionin ehdottaman tietosuoja-asetuksen päämäärä on luoda organisaatioissa kulttuuri, jossa henkilötietojen käsittelyä ja suojaamista huomioidaan laajemmin ja perusteellisemmin kuin nykypäivänä. Organisaatiot, jotka jo hyvissä ajoin ottavat asetuksen vaatimat asiat huomioon, ovat kilpailussa etulyöntiasemassa.

2013-08-30

clojuTRE 2013

We attended clojuTRE 2013 this week and it was brilliant, thanks to Metosin and Nitor for organizing.

The presentations:

  • ZenRobotics, who can probably claim to have deployed Clojure into production for the longest here in Finland, had Jouni Seppänen and Joel Kaasinen talk about the challenges with resource handling and how to integrate with OpenCV.
  • Tero Parviainen gave a talk on Pedestal, an über hipster web development framework by Relevance. Tero's opinion was that it is still evolving and probably not something you would use for customer projects just yet.
  • International Clojure superstars Sam Aaron, Christophe Grand, Edmund Jackson and Meikel Brandmeyer from Lambda Next gave a presentation on core.async. The integration of a technical presentation with a comedy act was hilarious. For this topic it worked perfectly, even though there were a few deadlocks.
  • Last but not least was a weapons demonstration by Jari Länsiö from Metosin. This super enthusiastic performance showed how to control an (almost) military grade drone with Clojure.

All in all the presentations were excellent and easily matched those at more established conferences.

After-party

After the official program most of us headed to the after-party. It was very interesting to share thoughts with other Finnish Clojurians on how to convince customers that Clojure, and its ecosystem, is production ready.

Final thoughts

We deployed our first Clojure project over a year ago and we have luckily been successful in proving our customers that Clojure is a viable option so that the majority of our new projects are Clojure-based. We found that some of the best arguments for Clojure are development speed, clear paradigm and consistent style. The last two lessen vendor lock-in which is always a worry for the customer.

2013-03-28

Querydsl 3.0 in a nutshell

Last week we released Querydsl 3.0. The goal of this release was to streamline Querydsl usage. In this post I will highlight a few of the general changes and some JPA/SQL specific ones.

Naming changes

Naming of query and subquery classes has been unified into the following form Query classes:
Module before after
Collections ColQuery (interface) - ColQueryImpl CollQuery
JDO JDOQLQuery (interface) - JDOQLQueryImpl JDOQLQuery (interface) - JDOQuery
SQL SQLQuery (interface) - SQLQueryImpl SQLQuery

No changes in JPA, Mongodb, Lucene and Hibernate Search.

Subquery classes:
Module before after
JPA JPQLSubQuery - JPASubQuery - HibernateSubQuery JPQLSubQuery (interface) - JPASubQuery - HibernateSubQuery
SQL SQLSubQuery SQLSubQuery

https://github.com/mysema/querydsl/issues/186

Tuples as first-class citizens

We encountered the following pattern often in our code
query.list(new QTuple(...))
and decided to change the varargs list method to return a list of tuples instead of object arrays. Thus
List<Tuple> result = query.list(new QTuple(a, b, …)) 
calls can be refactored into
List<Tuple> result = query.list(a, b, …)
The same change has been performed for the other varargs projection methods of the Projectable interface.

https://github.com/mysema/querydsl/issues/296

Unified distinct usage

The following distinct usage has been deprecated
query.listDistinct(a, b, c)
The preferred usage is now
query.distinct().list(a, b, c)
Which is more in line with other modifiers that are applied to the query.

https://github.com/mysema/querydsl/issues/303

Deeper defaults for path initialization

Querydsl metaclasses are now initialized one level further, so now there are less risks of running into issues when deep paths are used. And if you want to be absolute sure that all paths are initialized you can always use the accessor option as documented here.

https://github.com/mysema/querydsl/issues/301

Improved iteration for JPA

The iterate methods in Querydsl JPA query classes have been optimized to utilize Hibernate and EclipseLink specific iteration support. For Hibernate ScrollableResults are used and for EclipseLink Cursors.

https://github.com/mysema/querydsl/issues/346

Batoo JPA support

Batoo JPA is now officially supported by Querydsl.

https://github.com/mysema/querydsl/issues/265

Roo support

The @RooJpaEntity annotation of the Spring Roo project is now officially supported via the com.mysema.query.apt.roo.RooAnnotationProcessor processor. The workaround of using GenericExporter is no longer necessary.

https://github.com/mysema/querydsl/issues/318

with -> on

The with method for Querydsl JPA queries has been replaced with on, to promote a more consistent line of keywords in the query interfaces.

https://github.com/mysema/querydsl/issues/299

Support for order by nulls first

Explicit null ordering is now supported in Querydsl SQL. The supported extensions are
query.orderBy(lastName.asc().nullsFirst())
and
query.orderBy(lastName.asc().nullsLast())     

https://github.com/mysema/querydsl/issues/174

Builder methods for SQLTemplates types

The previous method of customizing SQLTemplates subclasses was a little awkward and involved further subclassing like this
new H2Templates(true){{
    newLineToSingleSpace();
}}
Querydsl 3 promotes now the usage of builder classes for customization
H2Templates.builder()
           .quote()
           .newLineToSingleSpace()
           .build();

https://github.com/mysema/querydsl/issues/355

Conclusion

The full list of changes for the 3.0 release are in 3.0.0, 3.0.0.BETA2 and 3.0.0.BETA1.

The latest stable release of Querydsl is 3.1. It fixes some packaging issues of the 3.0 release: https://groups.google.com/forum/#!topic/querydsl/I562uKQlheE. Usage of the 3.1 release is recommended.

Happy querying!

2012-07-03

Modularizing JavaScript Code with Backbone.js and RequireJS

Lately we have been doing a lot of JavaScript development and have been very happy about the improvements in libraries and frameworks, long gone are the days of seeing JavaScript just as a necessary evil. Unfortunately there is still legacy code lying around even in our codebase.

As an example I present an application that was developed by us: Semantic.hri.fi, it provides a Linked Open Data view of www.aluesarjat.fi statistics. What we will be looking at is the SPARQL search page and mainly its source code.

Even with a quick glance it is easy to spot several issues which are nowadays handled with a different approach:

  • String prototype is monkey patched with a function that is only used once
  • HTML is constructed in code via string concatenation
  • Application is not properly layered due to rendering happening in back-end data fetches
  • Hard to tell which functions are managing which area of the screen
  • File is very long
  • Global namespace is polluted with several variables

Now that we have identified some of the problem spots, how would we fix them?

String prototype is monkey patched with a function that is only used once

Monkey patching causes compatibility issues and thus is not very sensible. Luckily an easy fix is just to use a function that accepts the string as an argument. Another option of course is to use an external library for handling string manipulation, such as Underscore.string.

HTML is constructed in code via string concatenation

Instead of manually constructing HTML like this:

//...
html.push("<thead><tr>");
for (var i = 0; i < vars.length; i++){
  html.push("<th>" + vars[i] + "</th>");
}       
html.push("</tr></thead>");
// ..

I would use a templating library such as Handlebars. With Handlebars we would have a template declared in its own file like this:

...
<thead>
  <tr>
    {{#each vars}}
    <th>{{this}}<th>
    {{/each}}
  </tr>
</thead>
...

And the code to populate it with:

var template = Handlebars.compile(tableTemplate);
$('#results').html(template({vars: vars}));

Application is not properly layered due to rendering happening in back-end data fetches

Instead of doing DOM manipulation inside an AJAX success callback like this:

success: function(data){
  var defaultNamespaces = [];
  var bindings = data.results.bindings;
  for (var i = 0; i < bindings.length; i++){
    var binding = bindings[i];
    namespaces[binding["prefix"].value] = binding["ns"].value;
    defaultNamespaces.push("PREFIX ", binding["prefix"].value, ": <", binding["ns"].value, "> </br>\n");
  }
  
  $("#namespaces").html(defaultNamespaces.join(""));
  
  init();
}

I would rather just fire an event to inform anyone who is interested that we are done loading. For this we use a common event aggregator called vent, which is just a JavaScript object with Backbone.Events mixed in. With vent in use I would change the above to something like this:

success: function(data) {
  vent.trigger('query:success', data.results.bindings);
}

This enables the success function to only care about retrieving the data. How the data is processed later on is left to the subscriber(s).

Hard to tell which functions are managing which area of the screen

Looking at the code I am having a really hard time understanding which functions are responsible for managing which areas of the screen, they should definitely be somehow bundled. A great way to do this is with Backbone.Views.

I would probably first create a SparqlView which is composed of SearchView (one on the left) and SavedQueriesView. After that I would probably split SearchView into two separate views QueryView and ResultsView. Below is a pseudocode example of the structure:

SparqlView
  SearchView
    QueryView
    ResultsView
  SavedQueriesView

Backbone.Views would probably be used for smaller elements as well, but these changes alone would give us a better understanding of the structure, plus the benefit of all the goodies Backbone.View provides.

File is very long

This can be of course fixed by splitting the file into separate files and then loading them separately. Luckily for us, we have already split the code into separate Backbone entities so moving them into their own files is really easy:

ViewFilename
SparqlViewjs/views/sparql.js
SearchViewjs/views/sparql/search.js
QueryViewjs/views/sparql/search/query.js
ResultsViewjs/views/sparql/search/results.js
SavedQueriesViewjs/views/sparql/saved_queries.js

When the subview has only one parent I find it a good practice to put the subview into a subdirectory with the name of their parent.

Global namespace is polluted with several variables

We have split the code into separate files, but have not really done anything about polluting the global namespace and our HTML file is full of script sources. I would bring in RequireJS to sort out this mess.

With RequireJS we declare only our application entry point in the HTML:

<script data-main="main" src="js/libs/require/require.js"></script>

data-main specifies which file is used as the entry point, in our case it is main.js and its contents is something like the following:

require(['app'], function(App) {
  App.initialize();
});

require is a function that takes an array as its first argument and a function as its second. The values in the array are RequireJS modules which are fed into the function as arguments. So in this case the argument App is module app.js. Inside the function we call App's method initialize, let's have a look at app.js:

define(['js/libs/jquery/jquery', 'js/views/sparql.js'],
  function($, SparqlView) {
  var initialize = function() {
    new SparqlView({el: $('#sparql')});
  }

  return {initialize: initialize};
});

Here we have defined a RequireJS module that has two dependencies: jQuery and SparqlView and we only expose a hash containing one function, initialize. When called, initialize creates a new SparqlView, SparqlView is defined as a RequireJS module as all of its subviews.

We now hopefully have a clear view on how to take old JavaScript code to the present.

2012-06-21

Form Data Extraction with Backbone.js, Underscore.js and jQuery

Our weapon of choice for building single-page web applications is Backbone.js. Unlike a few of its competitors, such as Ember.js, Knockout, Backbone does not support model binding out of the box. There are a some libraries that add model binding support to Backbone, probably first on the block was Backbone.ModelBinding which has now been discontinued, Backbone.ModelBinder and Backbone Bindings.

Even though these libraries make sense in some cases, we have noticed that usually when starting out, Backbone's default tools (Backbone itself, Underscore.js and jQuery) are enough to get the ball rolling, as demonstrated below.

HTML

<form>
  Name: <input type="text" name="name"/>
  Age: <input type="text" name="age"/>
  <input type="submit" value="Submit"></input>
</form>

JavaScript

var UserForm = Backbone.View.extend({
  events: {'submit': 'save'},

  initialize: function() {
    _.bindAll(this, 'save');
  },
  
  save: function() {
    var arr = this.$el.serializeArray();
    var data = _(arr).reduce(function(acc, field) {
      acc[field.name] = field.value;
      return acc;
    }, {});
    this.model.save(data);
    return false;
  }
});

var userForm = new UserForm({el: this.$('form'), model: new User()});

To understand the actual data extraction I will explain serializeArray and reduce.

serializeArray is a jQuery method that takes a form and constructs an array of all the input elements that have a name attribute. The resulting array is in the following form:

[{name: "name", value: "John Smith"}, ...]

An array of maps is not quite something we are looking for, so we have to do another transformation.

reduce is used to build up a new value from the array with an original seed value (empty map) and a function. The function's first argument acc is the return value of the previous function call and the second argument field is the current element in the array. Now we have something that can be passed to Backbone.Model.save:

{name: "John Smith", age: 34}

2012-03-09

Scalagen - Java to Scala Conversion

Scalagen was born out of the idea that we would like to port some of our projects from Java to Scala source code. Instead of porting all the code manually a tool could be used to do most of the bulk work. We did some initial experiments with Jatran, but found it lacking.

After that we investigated some options to write such a tool ourselves. First we needed a parser for Java sources to turn into abstract syntax trees. We decided to use the Javaparser framework since it supports Java 5 source and had an easy to use API.

Initial Architecture

Then work began on Scalagen. The initial parts were written in Java, such as a visitor implementation for the javaparser which prints out the tree as Scala syntax. After that had been done this visitor was used to convert the visitor class from Java to Scala. After that the implementation of Scalagen continued to be fully Scala-based.

After that we wrote visitor implementations which did transformations on the AST. Needed transformations were to extract static members into companion objects and to change various control statements into a more Scala-esque form. At first the visitor would just mutate the original AST from one state to another. This proved to cause problems with Scala's immutable collection types so we rewrote the transforming visitors to always return new detached AST versions.

Now the initial architecture was set up and further transformations could be written. After a while we had an impressive list of transformations which could be easily used via both their class and object variants:

object Primitives extends Primitives


class Primitives extends UnitTransformerBase {

...

}

Wrapping the Parser

Then we began to adapt the javaparser API to be more easy to use. We declared short type aliases in a singleton object to strip off redundant suffixes such as Stmt and Expr:

type Annotation = AnnotationExpr 

type AnnotationDecl = AnnotationDeclaration 

type AnnotationMember = AnnotationMemberDeclaration

type Assign = AssignExpr  

type Binary = BinaryExpr   

type Block = BlockStmt 

type BodyDecl = BodyDeclaration

Then we began to write deconstructors to make AST pattern matching more concise:

object FieldAccess {
  def unapply(f: FieldAccess) = Some(f.getScope, f.getField)
} 

object For {
  def unapply(f: For) = Some(toScalaList(f.getInit), f.getCompare, 
    toScalaList(f.getUpdate), extract(f.getBody))
}

object Foreach {
  def unapply(f: Foreach) = Some(f.getVariable, f.getIterable, 
    extract(f.getBody))
}

object MethodCall {
  def unapply(m: MethodCall) = Some(m.getScope, m.getName, 
    toScalaList(m.getArgs))
}

And something more complex which provides both a new-less constructor via apply and a deconstructor via unapply.

object VariableDeclaration {
  def apply(mod: Int, name: String, t: Type): VariableDeclaration = {
     val variable = new VariableDeclarator(new VariableDeclaratorId(name))
     new VariableDeclaration(mod, t, variable :: Nil)
   }
  def unapply(v: VariableDeclaration) = 
    Some(v.getType, toScalaList(v.getVars))
}

The singleton names matched the type aliases so we had a very Scala-like meta-layer on top of the javaparser AST classes.

Here is a fairly complex example of how the pattern matching could be used in the transforming visitors.

override def visit(nn: Foreach, arg: CompilationUnit): Node = {
  val n = super.visit(nn, arg).asInstanceOf[Foreach]
  n match {
    case Foreach(
      VariableDeclaration(t, v :: Nil),
      MethodCall(scope, "entrySet", Nil), body) => {
        val vid = v.getId.toString
        new Foreach(
          VariableDeclaration(0, "(key, value)", Type.Object),
          scope, n.getBody.accept(toKeyAndValue, vid).asInstanceOf[Block])    
    }
    case _ => n
  }   
}

This looks still quite cryptic if you are not familiar with the AST structure of javaparser, but for the ones familiar with structure of the AST, this is a fairly intuitive way to match AST patterns.

Packaging

Scalagen provides direct Maven support via a plugin. You can use it directly via the command line like this

mvn com.mysema.scalagen:scalagen-maven-plugin:0.1.3:main \
  -DtargetFolder=target/scala

and for test sources

mvn com.mysema.scalagen:scalagen-maven-plugin:0.1.3:test \
  -DtargetFolder=target/scala

Here is the snippet for an explicit configuration in a POM:

<plugin>
 <groupId>com.mysema.scalagen</groupId>
 <artifactId>scalagen-maven-plugin</artifactId>
 <version>0.1.3</version>
</plugin>

To convert main sources run

mvn scalagen:main

and to convert test sources run

mvn scalagen:test

The conversion results are to be seen as a starting point for the Java to Scala conversion. Some elements are not transformed correctly for various reasons and will need manual intervention.

Finally

Scalagen is an experimental effort and has still lots of rough edges, but we are open to improvement suggestions and stylistic changes.

2011-11-01

On Ontology creation

Last week I held a presentation at the Kirjastoverkkopäivät event in Helsinki and talked about the stages of Ontology creation. Instead of posting the slides here I decided to elaborate more on the topics of the presentation.
The need for ontology creation arises often from having to agree on a formal terminology. Formal terminologies are important to share understanding and to reduce ambiguity. And to create a formal terminology a conceptual framework is needed to build on. This is where ontologies come in.
An ontology is a formal representation of knowledge as a set of concepts within a domain, and the relationships between those concepts.

Ontology standards

To be able to share ontologies and to reuse other ontologies in fragments or full form a common agreement on the building blocks of ontologies and their meaning is needed. The Semantic Web effort of the World Web Foundation has been dealing with this goal and has published several standards to tackle the problem.
The most relevant ontology standards today are RDF Schema, OWL 1.0 & 2.0 and SKOS.
  • RDF Schema example
    published in 1998, standardized in 2004
  • OWL example
    • 1.0 (2004)
      divided into the dialects Light, DL (Description Logic) and Full, extends RDF Schema
    • 2.0 (2009)
      divided into the profiles EL, QL and RL
  • SKOS example
    ontology for vocabularies and thesauri
RDF Schema and OWL are ontology frameworks which provide a type system of classes, properties and instances and SKOS can be seen as a ontology which defines concept types and relations between concepts.
The following sections will show examples to illustrate the differences and similarities of the presented standards.

RDF Schema

The Wine ontology is a popular RDF Schema example as it presents in compact form a lot of RDF schema building blocks. The boxes of the images present classes, predicates and instances while the lines present properties. The ElyseZinfandel box for example is connected to the Zinfandel box via the rdf:type relation which means the resource ElyseZinfandel has the type Zinfandel. It is also connected to the Elyse box, but via the hasMaker relation, and to the Red box, via the hasColor relation.
The predicates rdfs:subClassOf and rdfs:subPropertyOf are used to declare subclass and subproperty relations. rdfs:domain declares the subject range of a property, the type this predicate applies to while rdfs:range declares the object range of a property, i.e. the value range of the property.

OWL

The BBC Programmes ontology defines various concepts of the Media domain. This excerpt highlights the the owl:disjointWith relation of the OWL format.The declaration ProgrammeItem owl:disjointWith Brand for example declares that ProgrammeItem instances cannot be Brand instances and vice versa. Both ProgrammeItem and Brand are sub classes of the Programme class.

SKOS

The magazine ontology of ONKI defines concepts to be used for categorization of magazines. For example Living has the Antiques subcategory while Antiques has the subcategory Antique stores. The skos:broader relation is comparable with the rdfs:subClassOf relation, but while rdfs:subClassOf is transitive, skos:broader isn't. For transitive relations the skos:broaderTransitive relation needs to be used.
For a more detailed presentation of the W3C Ontology and Vocabulary formats visit here.

Creation of your own ontology

Sometimes the available ontologies don't cover your problem domain or define semantics that are not compatible with your needs. Defining your own ontology is then a valid option.
Ontology creation begins of with informal sketching of the concepts of the problem domain. Paper and pen, mind maps or any other sketching tool can be used for this. After that the main modeling dimensions should be defined. Do the concepts define a type or categorization system? Are subclass or subconcept relations the dominating relations or maybe temporal ones or something else? The answers to these questions will guide you to the selection of the appropriate modeling standard. As mentioned before RDF Schema and OWL should be picked to model type systems and SKOS for categorizations.

Relation of your own ontology to others

Sometimes most of your problem domain has already been formalized appropriately in other ontologies. In such a case you can link your own ontology to these external ones. Your own classes, predicates and concepts can be specializations and generalizations of appropriate external entities. Another common way is to define equivalence between entities via the the owl:sameAs relation.
The owl:sameAs relation is used much in the Linked Open Data movement to connect heterogenous datasets.

Where to go next?

Maintenance, further development and distribution need to be taken into account as well. Having the facilities in house to tackle these aspects is a great benefit. Otherwise probably a public host for ontologies such as ONKI should be picked.

ONKI

ONKI is a central ontology host of the finnish Semantic Web scene and is maintained by the SeCo team of the Aalto University. ONKI hosts 72 ontologies and vocabularies altogether: 47 OWL ontologies, 2 RDF Schema ontologies and 22 SKOS vocabularies. Most of these are freely available with a liberal license. The SeCo provides support for the ONKI service via email and notifices via Twitter of service changes and problems.

Using an ONKI ontology

ONKI provides several tools to use ontologies :
  • ONKI Browser - a faceted browser to explore ontology content
  • SOAP interface - for service integration
  • ONKI Selector widget - a JavaScript widget for web application integration
  • Downloads - in RDF/XML form to use the data in your own applications

ONKI Browser

This screenshot shows a part of the ONKI Browser tool, a faceted search for the ONKI content. The ONKI Browser provides free text search for ontologies and concepts and provides a navigatable tree visualization of the concept inheritance.

SOAP interface

The following methods are available in the ONKI SOAP interface:
  • getAvailableLanguages - languages used in concept names in an ontology.
  • getAvailableTypeUris - supported concept type URIs of an ontology.
  • search - searching for ontological concepts.
  • getLabel - fetching a label for the concept with a given URI.
  • expandQuery - expanding concept(s) with given URI(s) (e.g. to subconcepts) for queries.
  • getProperties - fetching properties for the concept with a given URI.
  • getConceptTree - fetching concept hierarchy for the concept with a given URI.
A REST/JSON interface is in beta phase, but the interface is still subject to change.

ONKI Selector wdiget

The ONKI Selector widget is a JavaScript script which can easily be embedded into modern web applications. It provides autocomplete based selection of ontology concepts. It is a fairly easy integration, but provides only label and URI of the selected concept, super concepts need to be fetched separately.
The displayed form fields offer the selection of a specific ontology to be targeted in the search or all, the search query and the language of the labels.

Using an ONKI ontology in your own service

While ONKI is a general ontology host, most of the ontologies are connected to the central YSO ontology, a general finnish language ontology. The following picture shows the umbrella function of the YSO ontology:
The benefit of this approach is that when using an ONKI ontology you have a rich foundation to start with, the downside is that the amount of concepts is huge, the YSO ontology alone hosts 20.000 concepts. Avoiding clashes and semantic mismatches can be difficult with so many concepts to be taken into account.
As the background of YSO is YSA, a central Finnish language glossary, it should probably be treated more as a vocabulary than ontology.

Conclusion

This article presented an overview of the ontology standards RDF Schema, OWL and SKOS with examples and a small introduction into the toolset of the ONKI service. If you have experiences with the ONKI service you'd like to share please comment on this post.

2011-08-26

Accessing Helsinki region statistics from your webapp

In this post I'd like to describe how the statistical data from the Semantic.hfi.fi service can be accessed from your web application via the demo app Areastats. Semantic.hri.fi offers several JSON/P web services which can directly be used from any website.

Areastats displays statistics when accessed from within the Finnish capital region. It uses internally the

  • HTML 5 Geolocation API to obtain the access location as WGS 85 coordinates,
  • a simple webservice to convert the coordinates into a code for the area the location resides in,
  • the Semantic.hri.fi Faceted Search API to access statistics for this area and
  • the jQuery API for DOM manipulation and Ajax

The geo-location-js API is used as an adapter for the incompatible Geolocation APIs used in various browsers. It provides a convenient access point to register callbacks for action when the location could be obtained and when not:

if (geo_position_js.init()) {
    geo_position_js.getCurrentPosition(geo_success, geo_error);
}

The geo_success callback then retrieves the area code from a web service via Ajax and uses the fetch_stats function to process the result:

function geo_success(p) {
  $.ajax({
    url: "polygon", 
    data: { "latitude": p.coords.latitude, "longitude": p.coords.longitude}, 
    dataType: "json", 
    success: fetch_stats,
    error: geo_error
  });
}

The fetch_stats then makes an Ajax call to obtain the results:

function fetch_stats(d) {
  var alue = "alue:_" + d.KUNTA + "_" + d.PIEN + "_" + d.Nimi;
 
  $.ajax({
    url: "http://semantic.hri.fi/search",
    data: { 
      "include": ["items"], "limit":50, "offset":0, 
      "value": [
        alue, 
        "vuosi:_2010", 
        "yksikkö:Henkilöä", 
        "dataset:A025_HKI_Vakiluku1962"] },
    dataType: "jsonp",
    success: fetch_stats_success,
    error: fetch_stats_error
  });
  $("#location").html(d.Nimi);
}

The query defines that we are interested in statistical items from the year 2010, with the unit persons (Henkilöä) and from the dataset dataset:A025_HKI_Vakiluku1962. For details on how to use the service of Semantic.hri.fi, please read the embedded documentation.

The fetch_stats_success callback then has the final results and draws a pie chart based on them:

function fetch_stats_success(d) {
  var age_index = -1;
  for (var i = 0; i < d.headers.length; i++) {
    if (d.headers[i] == "dimension:Ikäryhmä") {
      age_index = i;
    }
  }

  for (var i = 0; i < d.items.length; i++) {
    var key = d.items[i].values[age_index];
    key = key.substring(9);
    $("#"+key).html(d.items[i].value);
  }

  drawpie();
  $("#stats").show();
}

The fetch_stats_success function iterates the headers of the result to find the index of the column we are interested in and the iterates over the result items and maps the column data into the DOM of the local HTML document. After that the drawpie function is invoked and the stats can be shown.

This demo is related to the Helsinki Region Infoshare project, which aims to provide data of the Finnish capital region in Linked Open Data form.

Feel free to contact us if you need more info on the demo or the Semantic.hri.fi services.

2011-08-12

Querydsl in the wild

From time to time we observe how the Querydsl framework is used outside of Mysema to find out how Querydsl meets the needs of our user base. In this blog post I'd like to share some Querydsl usage patterns we found.

Expression factory class

As an alternative to Expression construction based on the core primitives reocurring constraints can be bundled as factory methods grouped by the used root types.

This example shows Expression factory methods for the User domain type

package org.springframework.data.jpa.example.repository;

import org.joda.time.LocalDate;
import org.springframework.data.jpa.example.domain.QUser;

import com.mysema.query.types.expr.BooleanExpression;

class UserExpressions {

    public static BooleanExpression goldMember() {
        return adult().and(createdAtleastOneYearAgo());
    }
    
    public static BooleanExpression adult() {
        return QUser.user.age.goe(18);
    }
    
    public static BooleanExpression createdAtleastOneYearAgo() {
        return QUser.user.creationDate.loe(new LocalDate().minusYears(1));
    }
    
    public static BooleanExpression differentFrontThanLastName() {
        return QUser.user.firstname.eq(QUser.user.lastname).not();
    }
    
    public static BooleanExpression usernameLongerThanSix() {
        return QUser.user.username.length().gt(6);
    }
    
}

Source: GitHub

$ as variable name for default entity path

In cases where Querydsl is used in combination with the Repository pattern it can be convenient to use a general variable name for the default entity path to be used. The dollar-sign is a valid candidate for this:

package reevent.dao;

import com.mysema.query.jpa.JPQLQuery;
import org.springframework.stereotype.Repository;
import org.springframework.transaction.annotation.Transactional;
import reevent.domain.QUser;
import reevent.domain.User;

@Transactional
@Repository
public class UserDaoJpa extends EntityDaoBase<User> implements UserDao {
    private static final QUser $ = QUser.user;

    @Override
    public User findByUsername(String username) {
        return queryByUsername(username).uniqueResult($);
    }

    @Override
    public boolean usernameExists(String username) {
        return queryByUsername(username).exists();
    }

    private JPQLQuery queryByUsername(String username) {
        return query().from($).where($.username.eq(username));
    }
}

Source: GitHub

Drawbacks with this approach are that some code inspection tools might complain about the unconventional variable name usage and that the dollar-sign is also used by the Querydsl Alias-functionality.

Hamcrest integration

Querydsl expressions can also be used for other tasks beside querying. A fine example is Hamcrest integration:

package org.hamcrest;

import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.PathMatcher.valueOf;
import static org.junit.Assert.assertThat;

import org.hamcrest.domain.Address;
import org.hamcrest.domain.Customer;
import org.hamcrest.domain.QCustomer;
import org.junit.Test;

public class PathMatcherTest {

    @Test
    public void testPathEvaluate() {
        Customer customer = new Customer();
        Address address = new Address();
        address.setCity("bla");
        address.setStreet("blaway 142");
        customer.setAddress(address);

        assertThat(customer, 
            valueOf(QCustomer.customer.address.street, 
            equalTo("blaway 142")));
    }
    
}

Hamcrest matchers can be used in unit tests for complex assertion construction.

Jeroen van Schagen, the author of the Hamcrest example has bundled his Querydsl extensions into the querydsl-support module.

Supplying filter criteria to general find methods

This pattern is extensively used in the Spring Data repositories. Here is a simple example:

QCustomer customer = QCustomer.customer;
LocalDate today = new LocalDate();
BooleanExpression customerHasBirthday = customer.birthday.eq(today);
BooleanExpression isLongTermCustomer = 
    customer.createdAt.lt(today.minusYears(2));
customerRepository.findAll(customerHasBirthday.and(isLongTermCustomer));

Advanced Spring Data JPA – Specifications and Querydsl

If you have innovative Querydsl usage yourself, please share it with us!

2011-03-16

Julkinen data seminaari - To 7.4.2011, Helsinki

Hallitus on tehnyt periaatepäätöksen (3.3.2011) julkishallinnon digitaalisten tietoaineistojen saatavuudesta. Tietoaineistojen tulee olla avoimesti saatavilla ja uudelleenkäytettävissä yhtenäisin, selkein ja kaikille tasapuolisin ehdoin. Periaatepäätös koskee ensisijaisesti sellaisenaan luovutettavissa olevia julkisia aineistoja, joiden käsittelyä ei lainsäädäntö rajoita.

Mutta mitä se tarkoittaa käytännössä? Miten julkisia tietoaineistoja tarjotaan?

Julkinen data seminaari torstaina 7.4.2011 Aalto yliopiston Chydenina-talossa Helsingissä tarjoaa tietoa julkisten tietovarantojen avaamiseen liittyen eri näkökulmilta. Seminaari on tarkoitettu kaikille tiedon parissa työskenteleville asiantuntijoille ja päättäjille kuntien ja valtion tehtävissä.

Tilaisuudessa esitellään:

  • mitä Julkinen data on
  • miten valtion periaatepäätöksellä pyritään edistämään julkisen hallinnon tietovarantojen saatavuutta ja käyttöä.
  • konkreettisia esimerkkejä mitä avoimilla tietovarantojen avulla on saatu aikaiseksi meillä ja maailmalla.
  • Case Helsingin kaupungin aluesarjat-tilastotietojen avaaminen ja teknisen ratkaisun rakentaminen

Tilaisuuden ohjelma:

8:30 Rekisteröityminen ja aamukahvit

9:00 Avaus - missä mennään julkisten tietovarantojen avauksessa, Taru Rastas, Liikenne ja Viestintäministeriö

9:15 Julkinen data - mitä kaikkea se onkaan? Tapio Nurminen, FloApps Oy

10:00 Kahvitauko

10:15 Case esittely - Aluesarjat-tilastot avoimeksi, Pekka Vuori, Yliaktuaari, Helsingin Kaupungin Tietokeskus

10:45 Q&A + Aluesarjat Demot by FloApps & Mysema

11.30 Lounas

12.30 Julkista dataa hyödyntäviä esimerkkisovelluksia ja -palveluita Suomessa ja maailmalla. Tapio Nurminen, Toimitusjohtaja, FloApps Oy

13:00 Datan avaaminen käytännössä-teknisen ratkaisun rakentaminen, Samppa Saarela, Mysema Oy

13:30 Avoin tieto laadukkaaksi data managementin avulla, Nino Ilveskero, Talent Base Oy

14:00 Kahvitauko

14:15 Liiketoimintamallit julkisen datan avulla, Juho Lindman, Aalto Econ. CKIR

14:45-15:30 Yhteenveto + kysymykset - Tapio Nurminen ja Samppa Saarela

Tilaisuuden hinta on 75 € + alv. 23%

Ilmoittaudu ystävällisesti tilaisuuteen 25.3. mennessä kahvi- ja lounasjärjestelyjen johdosta ennakkoon osoitteessa:

http://bit.ly/h9Hk0K

Seminaari järjestetään: torstaina 7. huhtikuuta, klo 8.30 - 15.30 Aalto-Yliopiston Chydenia-talossa osoitteessa Runeberginkatu 22-24, Helsinki.

Järjestäjät ovat Flo Apps Oy ja Mysema Oy,
yhteistyössä Helsinki Regional Infoshare, Aalto Yliopiston Econ. osaston ja Talent Base Oy:n kanssa.

2011-01-21

Querying in SQL with Querydsl

Querydsl provides a typesafe querying layer on top of JPA, JDO, JDBC and other backends. This blog post presents a simple tutorial on how to get started with querying in SQL using Querydsl.

Querydsl for SQL provides a typesafe view of the relational schema and makes expressing database queries in Java as intuitive as possible.

Getting started


To get started with Querydsl for SQL using a Maven 2 based build environment, follow these steps.

Add the following dependencies to your Maven project and make sure that the Maven 2 repo of Mysema Source is accessible from your POM :

<dependency>
  <groupId>com.mysema.querydsl</groupId>
  <artifactId>querydsl-sql</artifactId>
  <version>${querydsl.version}</version>
</dependency>

<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-log4j12</artifactId>
  <version>1.6.1</version>
</dependency>   

The snippet to include the Mysema Source Maven repository is :

<repository>
  <id>msource</id>
  <url>http://source.mysema.com/maven2/releases/</url>
  <releases>
    <enabled>true</enabled>
  </releases>
  <snapshots>
    <enabled>false</enabled>
  </snapshots>
</repository>

Set the variable querydsl.version to the latest Querydsl version available.

If you are not using Maven you can download the latest builds from here

Creating the Querydsl query types


To get started with creating Querydsl SQL queries you need to create Java classes for your relational schema. This can be done either in Java, via Maven or via Ant.

In Java


To get started export your schema into Querydsl query types like this :

java.sql.Connection conn = ...; 
MetaDataExporter exporter = new MetaDataExporter();
exporter.setPackageName("com.myproject.mydomain");
exporter.setTargetFolder(new File("src/main/java"));
exporter.export(conn.getMetaData());  

This declares that the database schema is to be mirrored into the com.myproject.domain package in the src/main/java folder.

The generated types have the table name transformed to mixed case as the class name and a similar mixed case transformation applied to the columns which are available as property paths in the query type.

In Maven


This functionality is also available as a Maven plugin. The presented example can be declared like this in the POM :

<plugin>
  <groupId>com.mysema.querydsl</groupId>
  <artifactId>querydsl-maven-plugin</artifactId>
  <version>${querydsl.version}</version>
  <executions>
    <execution>
      <goals>
        <goal>export</goal>
      </goals>
    </execution>            
  </executions>
  <configuration>
    <jdbcDriver>org.apache.derby.jdbc.EmbeddedDriver</jdbcDriver>
    <jdbcUrl>jdbc:derby:target/demoDB;create=true</jdbcUrl>
    <packageName>com.myproject.domain</packageName>
    <targetFolder>${project.basedir}/target/generated-sources/java</targetFolder> 
  </configuration>
  <dependencies>
    <dependency>
      <groupId>org.apache.derby</groupId>
      <artifactId>derby</artifactId>
      <version>${derby.version}</version>
    </dependency>
  </dependencies>
</plugin>  

Use the goal test-export to add the targetFolder as a test compile source root instead of a compile source root.

In ANT


The ANT task com.mysema.query.sql.ant.AntMetaDataExporter of the querydsl-sql module provides the same functionality as an ANT task. The configuration parameters of the task are jdbcDriverClass, dbUrl, dbUserName, dbPassword, namePrefix, targetPackage, targetSourceFolder, schemaPattern, tableNamePattern, exportBeans and innerClassesForKeys.

Using query types


To create queries with Querydsl you need to instantiate variables and Query implementations. We will start with the variables.

For each table in your mirrored schema a query type is created. For example for the table customer Querydsl will generate a query type with the simple name QCustomer into the folder and package declared in the code generation setup. QCustomer can be used as a statically typed variable in Querydsl queries as a representative for the customer table.

QCustomer has a default instance variable which can be accessed as a static field :

QCustomer customer = QCustomer.customer;

Alternatively you can define your own QCustomer instances like this :

QCustomer customer = new QCustomer("c");


Querying


Querying with Querydsl SQL is as simple as this :

QCustomer customer = new QCustomer("c"); // alias for the CUSTOMER table

SQLTemplates dialect = new HSQLDBTemplates(); // SQL-dialect
SQLQuery query = new SQLQueryImpl(connection, dialect); 
List<String> lastNames = query.from(customer)
    .where(customer.firstName.eq("Bob"))
    .list(customer.lastName);

which is transformed into the following sql query, assuming that the related table name is customer and the columns first_name and last_name :

SELECT c.last_name 
FROM customer c 
WHERE c.first_name = 'Bob'

Internally Querydsl SQL uses prepared statements.

General usage


Use the the cascading methods of the com.mysema.query.sql.SQLQuery interface like this

from : Define the query sources here.

innerJoin, join, leftJoin, fullJoin, on : Define join elements using these constructs. For the join methods the first argument is the join source and the second the target (alias).

where : Define the query filters, either in varargs form separated via commas or cascaded via the and-operator.

groupBy : Define the group by arguments in varargs form.

having : Define the having filter of the "group by" grouping as an varags array of Predicate expressions.

orderBy : Define the ordering of the result as an varargs array of order expressions. Use asc() and desc() on numeric, string and other comparable expression to access the OrderSpecifier instances.

limit, offset, restrict : Define the paging of the result. Limit for max results, offset for skipping rows and restrict for defining both in one call.

Ordering


The syntax for declaring ordering is

query.from(customer)
    .orderBy(customer.lastName.asc(), customer.firstName.asc())
    .list(customer.firstName, customer.lastName);

which is equivalent to the following native SQL

SELECT c.first_name, c.last_name
FROM customer c 
ORDER BY c.last_name ASC, c.first_name ASC

Grouping


Grouping can be done in the following form

query.from(customer)
    .groupBy(customer.lastName)
    .list(customer.lastName);

which is equivalent to the following native SQL

SELECT c.last_name 
FROM customer c 
GROUP BY c.last_name

Subqueries


To create a subquery you create a com.mysema.query.sql.SQLSubQuery instance, define the query parameters via from, where etc and use unique or list to create a subquery, which is just a type-safe Querydsl expression for the query. unique is used for a unique (single) result and list for a list result.

query.from(customer).where(
  customer.status.eq(new SQLSubQuery().from(customer2).unique(customer2.status.max()))
  .list(customer.all())

Another example

query.from(customer).where(
  customer.status.in(new SQLSubQuery().from(status).where(status.level.lt(3)).list(status.id))
  .list(customer.all())

Query extension support


Custom query extensions to support engine specific syntax can be created by subclassing com.mysema.query.sql.AbstractSQLQuery and adding flagging methods like in the given MySQLQuery example :

public class MySQLQuery extends AbstractSQLQuery<MySQLQuery>{
    
    public MySQLQuery(Connection conn) {
        this(conn, new MySQLTemplates(), new DefaultQueryMetadata());
    }
    
    public MySQLQuery(Connection conn, SQLTemplates templates) {
        this(conn, templates, new DefaultQueryMetadata());
    }
    
    protected MySQLQuery(Connection conn, SQLTemplates templates, QueryMetadata metadata) {
        super(conn, new Configuration(templates), metadata);
    }
    
    public MySQLQuery bigResult(){
        return addFlag(Position.AFTER_SELECT, "SQL_BIG_RESULT ");
    }
    
    public MySQLQuery bufferResult(){
        return addFlag(Position.AFTER_SELECT, "SQL_BUFFER_RESULT ");
    }
 
 
    // ...   
}        


The flags are custom SQL snippets that can be inserted at specific points in the serialization. The supported positions are the enums of the com.mysema.query.QueryFlag.Position enum class.

Using DDL commands


CREATE TABLE commands can be used in fluent form via the com.mysema.query.sql.ddl.CreateTableClause class. Here are some examples :

new CreateTableClause(conn, templates, "language")
 .column("id", Integer.class).notNull()
 .column("text", String.class).size(256).notNull()
 .primaryKey("PK_LANGUAGE","id")
 .execute();

new CreateTableClause(conn, templates, "symbol")
 .column("id", Long.class).notNull()
 .column("lexical", String.class).size(1024).notNull()
 .column("datatype", Long.class)
 .column("lang", Integer.class)
 .column("intval",Long.class)
 .column("floatval",Double.class)
 .column("datetimeval",Timestamp.class)
 .primaryKey("PK_SYMBOL","id")
 .foreignKey("FK_LANG","lang").references("language","id")
 .execute();

new CreateTableClause(conn, templates, "statement")
 .column("model", Long.class)
 .column("subject", Long.class).notNull()
 .column("predicate", Long.class).notNull()
 .column("object", Long.class).notNull()
 .foreignKey("FK_MODEL","model").references("symbol","id")
 .foreignKey("FK_SUBJECT","subject").references("symbol","id")
 .foreignKey("FK_PREDICATE","predicate").references("symbol","id")
 .foreignKey("FK_OBJECT","object").references("symbol","id")
 .execute();    

The constructor of CreateTableClause takes the connection, the templates and the table name. The rest is declared via column, primaryKey and foreignKey invocations.

Here are the corresponding CREATE TABLE clauses as they are executed.

CREATE TABLE language (
  id INTEGER NOT NULL,
  text VARCHAR(256) NOT NULL,
  CONSTRAINT PK_LANGUAGE PRIMARY KEY(id)
)

 CREATE TABLE symbol (
  id BIGINT NOT NULL,
  lexical VARCHAR(1024) NOT NULL,
  datatype BIGINT,
  lang INTEGER,
  intval BIGINT,
  floatval DOUBLE,
  datetimeval TIMESTAMP,
  CONSTRAINT PK_SYMBOL PRIMARY KEY(id),
  CONSTRAINT FK_LANG FOREIGN KEY(lang) REFERENCES language(id)
)

 CREATE TABLE statement (
  model BIGINT,
  subject BIGINT NOT NULL,
  predicate BIGINT NOT NULL,
  object BIGINT NOT NULL,
  CONSTRAINT FK_MODEL FOREIGN KEY(model) REFERENCES symbol(id),
  CONSTRAINT FK_SUBJECT FOREIGN KEY(subject) REFERENCES symbol(id),
  CONSTRAINT FK_PREDICATE FOREIGN KEY(predicate) REFERENCES symbol(id),
  CONSTRAINT FK_OBJECT FOREIGN KEY(object) REFERENCES symbol(id)
)  

Using DML commands


All the com.mysema.query.dml.DMLClause implementations in the Querydsl SQL module take three parameters, the java.sql.Connection instance, the com.mysema.query.sql.SQLTemplates instance used in the queries and the main entity the DMLClause is bound to.

Insert examples :

// with columns
new SQLInsertClause(conn, dialect, survey)
    .columns(survey.id, survey.name)
    .values(3, "Hello").execute();

// without columns
new SQLInsertClause(conn, dialect, survey)
    .values(4, "Hello").execute();

// with subquery
new SQLInsertClause(conn, dialect, survey)
    .columns(survey.id, survey.name)
    .select(new SQLSubQuery().from(survey2).list(survey2.id.add(1), survey2.name))
    .execute();

// with subquery, without columns
new SQLInsertClause(conn, dialect, survey)
    .select(new SQLSubQuery().from(survey2).list(survey2.id.add(10), survey2.name))
    .execute();

Update examples :

// update with where
new SQLUpdateClause(conn, dialect, survey)
    .where(survey.name.eq("XXX"))
    .set(survey.name, "S")
    .execute();
  
// update without where
new SQLUpdateClause(conn, dialect, survey)
    .set(survey.name, "S")
    .execute()  

Delete examples :

// delete with where
new SQLDelecteClause(conn, dialect, survey)
    .where(survey.name.eq("XXX"))
    .execute();
  
// delete without where
new SQLDelecteClause(conn, dialect, survey)
    .execute()  

Bean class generation


To create JavaBean DTO types for the tables of your schema use the com.mysema.query.sql.MetaDataExporter like this :

java.sql.Connection conn = ...; 
MetaDataExporter exporter = new MetaDataExporter();
exporter.setPackageName("com.myproject.mydomain");
exporter.setTargetFolder(new File("src/main/java"));       
exporter.setBeanSerializer(new BeanSerializer());
exporter.export(conn.getMetaData());  

Now you can use the bean types as arguments to the populate method in DML clauses and you can project directly to bean types in queries. Here is a simple example :

@Test
QEmployee e = new QEmployee("e");
           
// Insert
Employee employee = new Employee();
employee.setFirstname("John");
Integer id = insert(e).populate(employee).executeWithKey(e.id);
employee.setId(id);

// Update
employee.setLastname("Smith");
assertEquals(1l, update(e).populate(employee).where(e.id.eq(employee.getId())).execute());

// Query
Employee smith = query().from(e).where(e.lastname.eq("Smith")).uniqueResult(e);
assertEquals("John", smith.getFirstname());
        
// Delete 
assertEquals(1l, delete(e).where(e.id.eq(employee.getId())).execute());

Where to go next


See the related section of the latest Reference docs for more information.

2010-11-10

MongoDB with Querydsl

Querydsl provides a typesafe querying layer on top of JPA, JDO, JDBC and other backends. This blog post presents a simple tutorial on how to get started with querying on MongoDB using Querydsl.

Maven integration


To get started with Querydsl for MongoDB using a Maven 2 based build environment, follow the following steps.

Add the following dependencies to your Maven project and make sure that the Maven 2 repo of Mysema Source is accessible from your POM :

<dependency>
  <groupId>com.mysema.querydsl</groupId>
  <artifactId>querydsl-mongodb</artifactId>
  <version>${querydsl.version}</version>
</dependency>

<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-log4j12</artifactId>
  <version>1.6.1</version>
</dependency>   

Querydsl uses the Annotation Processing Tool of Java 6 for code generation which needs to be configured as well :

<project>
  <build>
  <plugins>
    ...
    <plugin>
      <groupId>com.mysema.maven</groupId>
      <artifactId>maven-apt-plugin</artifactId>
      <version>1.0</version>
      <executions>
        <execution>
          <goals>
            <goal>process</goal>
          </goals>
          <configuration>
            <outputDirectory>${generatedSources}<outputDirectory>
            <processor>com.mysema.query.mongodb.morphia.MorphiaAnnotationProcessor</processor>
          </configuration>
        </execution>
      </executions>
    </plugin>
    ...
  </plugins>
  </build>
</project>

The MorphiaAnnotationProcessor finds domain types annotated with the com.google.code.morphia.annotations.Entity and com.google.code.morphia.annotations.Embedded annotations and generates query types for them.

Run clean install and you will get your Query types generated into target/generated-sources/java.

If you use Eclipse, run mvn eclipse:eclipse to update your Eclipse project to include target/generated-sources/java as a source folder.

Now you are able to construct Querydsl query instances and instances of the query domain model.


Ant integration


If you have an Ant based build environment you can use one of the packaged releases from the Downloads page.

Place the jar files from the full-deps bundle on your classpath and use the following tasks for Querydsl code generation :

<!-- APT based code generation -->
   <javac srcdir="${src}" classpathref="cp">
     <compilerarg value="-proc:only"/>      
     <compilerarg value="-processor"/>
     <compilerarg value="com.mysema.query.mongodb.morphia.MorphiaAnnotationProcessor"/>
     <compilerarg value="-s"/>
     <compilerarg value="${generated}"/>
   </javac>
    
   <!-- compilation -->
   <javac classpathref="cp" destdir="${build}">      
     <src path="${src}"/>
     <src path="${generated}"/>
   </javac> 

Replace src with your main source folder, generated with your folder for generated sources and build with your target folder.

Using query types


To create queries with Querydsl you need to instantiate variables and Query implementations. We will start with the variables.

Let's assume that your project has the following domain type :

@Entity
public class Customer {

    private @Id ObjectId id;

    private String firstName;
    
    private String lastName;

    public String getFirstName(){
        return firstName;
    }

    public String getLastName(){
        return lastName;
    }

    public void setFirstName(String fn){
        firstName = fn;
    }

    public void setLastName(String ln)[
        lastName = ln;
    }

    public ObjectId getId() {
        return id;
    }

    public void setId(ObjectId id) {
        this.id = id;
    }
}

Querydsl will generate a query type with the simple name QCustomer into the same package as Customer. QCustomer can be used as a statically typed variable in Querydsl queries as a representative for the Customer type.

QCustomer has a default instance variable which can be accessed as a static field :

QCustomer customer = QCustomer.customer;

Alternatively you can define your own Customer variables like this :

QCustomer customer = new QCustomer("myCustomer");

Querying


For the MongoDB-module MorphiaQuery is the main Query implementation. It is instantiated like this :

// test database
String dbname = "testdb"; 
// Morphia configuration
Morphia morphia = new Morphia().map(Customer.class); 
// datastore
Datastore ds = morphia.createDatastore(dbname); 
QCustomer customer = QCustomer.customer;
MorphiaQuery<Customer> query = new MorphiaQuery<Customer>(morphia, ds, customer);

To retrieve the customer with the first name Bob you would construct a query like this :

QCustomer customer = QCustomer.customer;
MorphiaQuery<Customer> query = new MorphiaQuery<Customer>(morphia, ds, customer);
Customer bob = query.where(customer.firstName.eq("Bob"))
  .uniqueResult();

The where part defines the filter and uniqueResult tells Querydsl to return a single element. Easy, right?

And to use multiple filters use it like this

query
    .where(customer.firstName.eq("Bob"), customer.lastName.eq("Wilson"));   

Or like this

query
    .where(
      customer.firstName.eq("Bob").and(customer.lastName.eq("Wilson")));


General usage


Use the the cascading methods of the MorphiaQuery class like this

  • where : Define the query filters, either in varargs form separated via commas or cascaded via the and-operator.
  • orderBy : Define the ordering of the result as an varargs array of order expressions. Use asc() and desc() on numeric, string and other comparable expression to access the OrderSpecifier instances.
  • limit, offset, restrict : Define the paging of the result. Limit for max results, offset for skipping rows and restrict for defining both in one call.

Ordering


The syntax for declaring ordering is

query
    .orderBy(customer.lastName.asc(), customer.firstName.desc())
    .list();


Where to go next


For an up-to-date guide on how to use Querydsl with MongoDB, see the latest Reference Documentation.

And if you have any issues, questions or ideas for enhancement post a thread in the Mysema Source Forum.

Happy querying!