tag:blogger.com,1999:blog-89542772666831326562024-03-13T04:08:42.413+02:00Mysema BlogMysema on Java development, OpenSource and Semantic WebTimo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.comBlogger31125tag:blogger.com,1999:blog-8954277266683132656.post-47053634639611365892013-11-28T09:19:00.000+02:002013-11-28T09:19:58.297+02:00Querydsl user survey results<p>A few weeks ago we opened the <a href="https://www.surveyplanet.com/survey/6d73740b19f63db08033c07657367d39">Querydsl user survey</a> to gather some information on how Querydsl is used and what direction it would make sense to take it. Some of the results were a little surprising, but mostly the results confirmed that the decisions we had made in the past were mostly right and that most of ours users are extremely happy with the product.</p>
<p>Here is a summary of the results, the amounts are in percentages:</p>
<h3>How have you heard of Querydsl?</h3>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzzumvLcBfIzXpDpY6VEsSLEwmjiTX-zhP_dNTWWrAVchdEsOgvHw9uBWrCemzV1z45TJo5bYrAMpMKkgUCo2uI9bLbD0QMxXRnTZnId2eJgqD8OCbe-dsXgVACrh019FzR6ETS2Bnny8/s1600/querydsl_1.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzzumvLcBfIzXpDpY6VEsSLEwmjiTX-zhP_dNTWWrAVchdEsOgvHw9uBWrCemzV1z45TJo5bYrAMpMKkgUCo2uI9bLbD0QMxXRnTZnId2eJgqD8OCbe-dsXgVACrh019FzR6ETS2Bnny8/s1600/querydsl_1.png" /></a>
<h3>Which IDEs are you using?</h3>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1Z46fGo3R0jY1t30sKy_b1IcFmISZOF8HnUl9FmBYDtrKNwcEobwMEWfm5DvMQujRiSUwE7XfkzQidhmcaaAlVj08qQIxUP4N3H5Roxu1gcjRy1Mwzw0nwbpWdCR2lb6fA1Ed4kqsBqE/s1600/querydsl_2.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1Z46fGo3R0jY1t30sKy_b1IcFmISZOF8HnUl9FmBYDtrKNwcEobwMEWfm5DvMQujRiSUwE7XfkzQidhmcaaAlVj08qQIxUP4N3H5Roxu1gcjRy1Mwzw0nwbpWdCR2lb6fA1Ed4kqsBqE/s1600/querydsl_2.png" /></a>
<h3>Which build tools are you using with Querydsl?</h3>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNdLWqbyZ4QttEY8p0pldTKD7t2XDdDB2bVtOKpQkjeu8Zp5B8DPxrP4pP_hml3knRppgdTf_9_OQmIUm2AFzJma8tcnZWmcNZOOnMnVmqDeTVIN7SMYv0t4Sh1hfu-DPG95Sh6znOnTw/s1600/querydsl_3.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNdLWqbyZ4QttEY8p0pldTKD7t2XDdDB2bVtOKpQkjeu8Zp5B8DPxrP4pP_hml3knRppgdTf_9_OQmIUm2AFzJma8tcnZWmcNZOOnMnVmqDeTVIN7SMYv0t4Sh1hfu-DPG95Sh6znOnTw/s1600/querydsl_3.png" /></a>
<h3>Which Querydsl modules are you using?</h3>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXoC5NMuD9Q9w134I49dNHp9jcSnB4xwxO0zBnk2k971ODZpmtpMEhw8-BwzsF1spSfPVq666plmb9Uzh5VIpfg7XGTb4OetqSELt9YklsP2R0w7TexS5L2X0YAnc-5ZfH0v84WPiW0pc/s1600/querydsl_4.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXoC5NMuD9Q9w134I49dNHp9jcSnB4xwxO0zBnk2k971ODZpmtpMEhw8-BwzsF1spSfPVq666plmb9Uzh5VIpfg7XGTb4OetqSELt9YklsP2R0w7TexS5L2X0YAnc-5ZfH0v84WPiW0pc/s1600/querydsl_4.png" /></a>
<h3>What other databases would you like to see supported in Querydsl?</h3>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeKpyoYn5jY2Cagdr3lVOga_7WMdGppjywmH0TsIegdgvWmIsoMGc5r2i4SQO5tjgJi6DTU70iN4EF_yznxVcgJM8ULKxDW_DR1mJ6gBa1O1KHZCyvP_FvCN4GI0Wy4jkdsX3IgsVlfGc/s1600/querydsl_5.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeKpyoYn5jY2Cagdr3lVOga_7WMdGppjywmH0TsIegdgvWmIsoMGc5r2i4SQO5tjgJi6DTU70iN4EF_yznxVcgJM8ULKxDW_DR1mJ6gBa1O1KHZCyvP_FvCN4GI0Wy4jkdsX3IgsVlfGc/s1600/querydsl_5.png" /></a>
<h3>What JDK version/s are you using?</h3>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgw7bKh5TTD65CqHT0swSU2D4RnUwONjSGoej9Zn54QKyF937qx8-boNyBncgl0HkEZmrnobOWhh_P0HVTcj0KM3MFCVhPY_bBmxNSKZ9bvlPh0OG2npk1OwZB8Jd98XrZzm2ShqtIrWzY/s1600/querydsl_6.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgw7bKh5TTD65CqHT0swSU2D4RnUwONjSGoej9Zn54QKyF937qx8-boNyBncgl0HkEZmrnobOWhh_P0HVTcj0KM3MFCVhPY_bBmxNSKZ9bvlPh0OG2npk1OwZB8Jd98XrZzm2ShqtIrWzY/s1600/querydsl_6.png" /></a>
<h3>What is your overall satisfaction with Querydsl?</h3>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFCM9CHiv2TpmGMUohFcHeehc_reHoHey_WTROFEN3E-_qrnsV8slzq2-YvW14Cq-SO1dbnKsxqoPhUgsp9EKbY6hE0SBtiSVjktolK2B6GK-9Oehzb8nSTA6YEYliuq5vf-u22-bM_Tw/s1600/querydsl_7.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFCM9CHiv2TpmGMUohFcHeehc_reHoHey_WTROFEN3E-_qrnsV8slzq2-YvW14Cq-SO1dbnKsxqoPhUgsp9EKbY6hE0SBtiSVjktolK2B6GK-9Oehzb8nSTA6YEYliuq5vf-u22-bM_Tw/s1600/querydsl_7.png" /></a>
<h3>What are the major benefits in using Querydsl?</h3>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-KaeN-ztW6yJ9a9R80te9yS865Inkck0dZV6oNHKdR7fxOZUZShTxtzQ8xYa5Gj0eyBHb1CCqqaOEjfxUKhoN18k-odsXEf5XQ9KveoT0TrOK8K2y2tYVSyMQ5OV5IlARdLcQaJ6ygrM/s1600/querydsl_8.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-KaeN-ztW6yJ9a9R80te9yS865Inkck0dZV6oNHKdR7fxOZUZShTxtzQ8xYa5Gj0eyBHb1CCqqaOEjfxUKhoN18k-odsXEf5XQ9KveoT0TrOK8K2y2tYVSyMQ5OV5IlARdLcQaJ6ygrM/s1600/querydsl_8.png" /></a>
<h3>How have you participated in the Querydsl community?</h3>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgABeIDZgyYotARt7tE-4SmGel6M4ktwDQrjTHhsEcZiOBySdLmIt2e0ODWzK1Z0sG7naB3fVizS9wHXoZ6inSyz0SXqnU320ul27K7ENHVbI2yTpiALRfd_yzUg-zkFbDjJLV12rjt-DI/s1600/querydsl_9.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgABeIDZgyYotARt7tE-4SmGel6M4ktwDQrjTHhsEcZiOBySdLmIt2e0ODWzK1Z0sG7naB3fVizS9wHXoZ6inSyz0SXqnU320ul27K7ENHVbI2yTpiALRfd_yzUg-zkFbDjJLV12rjt-DI/s1600/querydsl_9.png" /></a>
<h3>What commercial Querydsl services would you be interested to try?</h3>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbCIVUdvlttdkXlL3yVygnh5Cn5jWPBMQMIiU-d7kcKeFGnHHhdvRpG3cm5AhN9M66dhRWSDHJ9t8lDfV2FRf02gLVD_CmYhp2ajzk4u3mSw98zc7eikTMkm8t1z3IJER8hGIs_lneH80/s1600/querydsl_10.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbCIVUdvlttdkXlL3yVygnh5Cn5jWPBMQMIiU-d7kcKeFGnHHhdvRpG3cm5AhN9M66dhRWSDHJ9t8lDfV2FRf02gLVD_CmYhp2ajzk4u3mSw98zc7eikTMkm8t1z3IJER8hGIs_lneH80/s1600/querydsl_10.png" /></a>
<h3>Do you have anything else to tell us</h3>
<p>This question was not multiple-choice, so only the most common topics are listed below</p>
<ul>
<li>Better docs, more examples</li>
<li>Roadmap of coming features</li>
<li>Donation page</li>
<li>Better readability for Querydsl SQL queries</li>
<li>Dedicated Gradle support</li>
<li>More advanced SQL functionality</li>
<li>Continue with the current licensing model</li>
<li>Better control of code generation target packages</li>
<li>Better docs for Hibernate Search and Collections modules</li>
</ul>
<h3>Conclusion</h3>
<p>Based on these promising results we will take the following actions:</p>
<ul>
<li>Expand our commercial offers</li>
<li>Invest into Querydsl SQL, since it is the second most popular Querydsl module</li>
<li>Improve the general documentation</li>
<li>Provide a public Roadmap for Querydsl development</li>
</ul>
<p>Feedback on Querydsl and its development model is of course also welcome in any other form. Let us know what features you are missing and what use cases should be supported better.</p>
Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com0tag:blogger.com,1999:blog-8954277266683132656.post-38129628149158154962013-11-21T11:48:00.000+02:002013-11-21T17:15:34.328+02:00Hibernate challenges: Entity persistence<p>Hibernate has been the most used persistence technology in our Java projects. We have optimized Hibernate usage via our own tooling such as Querydsl JPA for querying as well as best practices and conventions for other parts such as entity lifecycle handling.</p>
<p>In this blog post I’d like to address Hibernate issues related to entity persistence and loading. Some of these challenges we have faced ourselves in customer projects and others are issues our customers have come across.</p>
<h3>Usage of right session level methods </h3>
<p>Hibernate provides multiple methods for persisting entities with slightly different semantics. To understand the methods better it is good to understand the different states Hibernate managed entities go through.</p>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbxEO8eHH2qUcL-O6aaBA_LI9n_FDVBGhrAPCUgR5Srxe27kSUwUEg50Qo-quSC4hjJSI3ufObU7CRsgicshEhijCp1LEDN_F-tIdjG4EVk13VLGCdjIYNvE-6Bi9OedKfVvJ_djNfxO8/s1600/object-states.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbxEO8eHH2qUcL-O6aaBA_LI9n_FDVBGhrAPCUgR5Srxe27kSUwUEg50Qo-quSC4hjJSI3ufObU7CRsgicshEhijCp1LEDN_F-tIdjG4EVk13VLGCdjIYNvE-6Bi9OedKfVvJ_djNfxO8/s1600/object-states.png" width="600" height="300" /></a>
<p>The starting point for new entities is the <b>transient</b> state. Transient entities are unknown to any Hibernate session and don’t have any database identity attached. Transient instances can be persisted by <code>Session.save()</code> and <code>Session.saveOrUpdate()</code>. save will assign the database identity value to the entity’s id property, but might not do any SQL INSERTs. Session.save() should be seen as the way to register an entity with Hibernate.</p>
<p>The <b>persistent</b> state is the state loaded and queried entities are in. Also transient entities become persistent when saved via <code>Session.save()</code> or <code>saveOrUpdate()</code>. Persistent instances are tracked by the Hibernate session and changes are persisted to the database at session flush time. </p>
<p>Persistent entities become <b>detached</b> via <code>Session.evict()</code>, <code>Session.clear()</code> and <code>Session.close()</code> calls. Detached entities do have a database identity, but are not attached to a running entity. Detached entities are usually used when entities are used in the view rendering phase for web application with an active session or in long running conversions with multiple sessions.</p>
<p>Persistent entities can be also be deleted via <code>Session.delete()</code>. The deletions on the database level happen at session flush time.</p>
<p>There is also a path back from detached entities to persistent entities via <code>Session.saveOrUpdate()</code>, <code>Session.update()</code> and <code>Session.merge()</code>. </p>
<p>A common problem with Hibernate session usage is that methods are not used in the right way or unnecessary session level method calls are made. The following two cases are quite common examples of wrong API usage.</p>
<p>It is quite common to see method calls to persist entities which are known to be managed by Hibernate.</p>
<pre class="brush:java" name="code">Person person = new Person();
session.save(person);
// modifications to person
session.update(person);
</pre>
<p>In this case the update call in the end is a NOOP since the person instance is already persistent. </p>
<p>Another common antipattern is to control the session size with too fine grained calls. </p>
<pre class="brush:java" name="code">Person person = new Person();
// modifications to person
session.save(person);
session.flush();
session.clear(); // or session.evict(person)
</pre>
<p>The code above makes the person instance persistent, assigns and id, saves it to the database on flush and clears the session on clear.<br />
There are several problems with this approach</p>
<ul>
<li>JDBC batching on INSERT and UPDATE can’t be utilized, since flushes happen too often</li>
<li>other entities are also removed from the session which might break surrounding code, evict helps in this case</li>
<li>the person instance is detached after this call and that needs to be taken into account in it’s usage</li>
</ul>
<p>Explicit flush and evict calls are often necessary for batch operations, but for other cases implicit flushes should be used on transaction ends.</p>
<h3>Mixing session level and SQL level manipulations</h3>
<p>Hibernate provides the possibility to mix session level operations on entities with direct updates to the database via DML clauses. When these get mixed things get complicated</p>
<pre class="brush:java" name="code">
List<MCustomer> customers = session.createQuery("select c from Customer c").list();
String hql = "update Customer c set c.name = :newName where c.name = :oldName";
int updatedEntities = s.createQuery(hql)
.setString("newName", newName)
.setString("oldName", oldName)
.executeUpdate();
for (Customer customer : customers) {
// modifications to customers
}
</pre>
<p>In this case the modifications to the customers on the Java level override the modifications via the DML clause, even if other properties are modified. Also the updates executed via the update are not reflected in the second level cache.</p>
<p>A good solution to this problem is to keep DML and session level operation usage in different transactions. If that is not possible one should be very careful not to create inconsistent end results.</p>
<h3>Efficient batch inserts and updates</h3>
<p>The promoted approach to batch insert entities via session level methods is </p>
<pre class="brush:java" name="code">for (int i = 0; i < 100000; i++) {
Customer customer = new Customer(.....);
session.save(customer);
if (i % 20 == 0) { //20, same as the JDBC batch size
//flush a batch of inserts and release memory:
session.flush();
session.clear();
}
}
</pre>
<p>This is a good approach if used in an isolated session. If other things happen before and after this code in the same session a more secure approach could be</p>
<pre class="brush:java" name="code">Set<Customer> batch = new HashSet<Customer>();
for (int i = 0; i < 100000; i++) {
Customer customer = new Customer(.....);
session.save(customer);
batch.add(customer);
if (batch.size() == 20) { //20, same as the JDBC batch size
for (Customer c : batch) {
session.evict(c);
}
session.flush();
batch.clear();
}
}
</pre>
<p>The same applies for batch updates. Usage of session.clear() in a more complex context is always risky.</p>
<h3>Efficient loading</h3>
<p>Efficient loading of entities and collections involves tuning the loading behaviour when querying.</p>
<p>It is usually a good practice to favor query scope optimizations of loading and fetching instead using annotations extensively, because this gives you more flexibility and emphasizes use case specific optimizations instead of general behaviour.</p>
<p>Collections are by default loaded lazily but they can fetched in a single select via the fetch keyword</p>
<pre class="brush:sql" name="code">select c from Company c
left join fetch c.departments
</pre>
<p>As an annotation inside Company it could be declared like this</p>
<pre class="brush:java" name="code"> @Entity
public class Company {
@Fetch(FetchMode.Join)
private Set<department> departments;
}
</pre>
<p>It is usually good to define as annotations the desired behaviour for loading via session.get() or session.load() and via query flags the optimizations for loading of multiple entities via a query.</p>
<h3>Hibernate or JPA API?</h3>
<p>Usage of the JPA 2 API is usually promoted in favor of Hibernate’s own Session API. Even the Hibernate team itself pushes the adoption of the JPA API <a href="http://www.theserverside.com/news/2240186700/The-JPA-20-EntityManager-vs-the-Hibernate-Session-Which-one-to-use">http://www.theserverside.com/news/2240186700/The-JPA-20-EntityManager-vs-the-Hibernate-Session-Which-one-to-use</a>.</p>
<p>In one of our projects we migrated from Hibernate Session usage to JPA API and the biggest pain points were related to the methods related to persistence of entities and related cascading operations. Queries were easily ported since they were already handled by Querydsl which supports both APIs.</p>
<p>The main methods in Hibernate to persist objects are session.save() and session.update(). In JPA they are entityManager.persist() and entityManager.merge(). Hibernate provides additionally the saveOrUpdate convenience method, which will use either save or update depending on the state of the object. A saveOrUpdate() equivalent is not available in the JPA and the entityManager.merge() method is not a direct replacement for session.update(), so it is not a trivial job to replace saveOrUpdate() usage when migrating to the JPA API.</p>
<p>The challenges we faced were</p>
<ul>
<li>no saveOrUpdate() replacement available and own implementation proved to be not fully compatible with the previous behaviour</li>
<li>Hibernate cascade annotations didn’t work reliably anymore when EntityManager persistence methods were used.</li>
</ul>
<p>We ended up replacing the Hibernate cascade annotations with JPA cascades and replacing saveOrUpdate calls with a custom persistOrMerge method which used the JPA API. </p>
<p>Having a strong DAO / Service division helps with migrating from one persistence API to another, but since the transaction boundary is often on the Service layer the persistence API behaviour often leaks to the service layer.</p>
<p>For a more indepth discussion of this problem read this blog post <a href="http://blog.xebia.com/2009/03/23/jpa-implementation-patterns-saving-detached-entities/">http://blog.xebia.com/2009/03/23/jpa-implementation-patterns-saving-detached-entities/</a></p>Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com0tag:blogger.com,1999:blog-8954277266683132656.post-76685069318386485042013-10-31T09:54:00.002+02:002013-11-01T16:30:48.151+02:00Querydsl commercial offers<p>We are frequently asked about our future plans regarding Querydsl and in which direction we intend to take it commercially. With the recent announcements of the jOOQ project to change its licensing model from a pure OSS licensing model to dual licensing our users are wondering if we might follow suit.</p>
<p>We have no intention of changing the licensing model of Querydsl and will keep on using a pure open source license for the foreseeable future. We feel that middleware should be transparent, and open source is one of the best models to guarantee both transparency and an interactive development model.</p>
<p>Our commercial model for Querydsl is open source software and commercial services. Querydsl is free to use and you will get free public support for it, but for private support, bigger additions and closed source extensions we are available for consulting and development at reasonable hourly rates.</p>
<h3>Training</h3>
<p>Querydsl training is available in packages and can also be customized for special needs. The available packages are</p>
<h4>Introduction</h4>
<p>one day of training to get an overview of the general functionality of Querydsl and usage examples for its most popular modules JPA and SQL</p>
<h4>Querydsl & JPA/Hibernate </h4>
<p>two days of training to get an overview of Querydsl and how it is best used with JPA including best practices for service/DAO design with Querydsl</p>
<h4>Querydsl & SQL</h4>
<p>two days of training to get a general overview of Querydsl and how it is best used with JDBC/SQL </p>
<h3>Project development</h3>
<p>Development services for projects involving Querydsl and related technologies are also available. We have worked with small startups and big global corporations to provide solutions using open source Java and Web technologies.</p>
<p>See our <a href="http://www.mysema.com/#work">references</a> for some examples of projects we’ve done in the past.</p>
<p>In addition, we have consulted a major Finnish software company on Hibernate performance problems. And later given training on Hibernate and Querydsl usage for their development team in St. Petersburg.</p>
<h3>Extensions</h3>
<p>Querydsl extensions in both open source and closed source form can be provided by us via development projects.</p>
Examples of such extensions are
<ul>
<li>support for SQL/NoSQL technologies that are not yet supported by Querydsl (e.g. DB2 or Cassandra)</li>
<li>higher level abstractions for Querydsl that fit into your IT infrastructure</li>
<li>integration of Querydsl into other middleware frameworks or libraries</li>
</ul>
<p>Contact us at <a href="mailto:querydsl@mysema.com">querydsl@mysema.com</a> if any of these options sound attractive to you. And if you haven’t yet participated in the <a href="http://t.co/FySLA58Y82">Querydsl user survey</a>, please do so.</p>
Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com0tag:blogger.com,1999:blog-8954277266683132656.post-56556129634185679832013-10-07T09:45:00.002+03:002013-10-07T10:13:27.928+03:00HelMet integration for Linked EventsThe most recent addition to <a href="http://events.hubi.fi/">Linked Events</a>, the event information aggregator developed by Mysema, is the integration of events taking place at libraries throughout the Helsinki region. This information is kindly provided by <a href="http://www.helmet.fi/">HelMet</a>.<br />
<br />
Adding new sources of data to an existing system is always a challenge. For example, the format of the new data might be incompatible with the existing data. This is true for event information in particular since there isn’t any generally accepted format, although some work is being done in that direction, see for example <a href="http://schema.org/Event">schema.org</a>, <a href="http://www.iptc.org/cms/site/?channel=CH0112">EventsML</a> or <a href="http://citysdk.ist.utl.pt/">CitySDK</a>. Other difficulties can arise from incompatibilities at the API level - even though a machine-friendly API is provided, it isn’t necessarily cut out for all possible use cases that application developers might come up with.<br />
<br />
Open data has become a priority for the public sector in many countries. The idea is enticing - make your data open and let people innovate with it, developing applications that the public sector itself does not have the resources to implement. New York has <a href="http://nycbigapps.com/">NYC BigApps</a>, Helsinki <a href="http://dev.hel.fi/">loves developers</a> and there is even an <a href="http://apps4pirkanmaa.fi/">Apps4Pirkanmaa competition</a>, to name just a few. The HelMet case is a good example of a public organization’s willingness to expose data for public use - information about events taking place in HelMet’s network of libraries is provided using a machine-friendly API, which we decided to leverage in the Linked Events portal.<br />
<h3>
The challenges of open data</h3>
Despite the easy-to-use API, we still encountered a few problems while integrating our new data source, problems that probably are rather typical for developers trying to leverage open data. In our case, several vital pieces of information were missing from the event data that the API provides - location information, translations and so on. All of these issues would certainly be possible to fix, but it would probably take a long time for the data provider, a large public sector organization, to go through the hoops of procurement, IT development outsourcing and so forth. Because of time constraints, we ended up scraping a website for the missing pieces of information, an unnecessarily brittle step in what could otherwise have been a very simple integration.<br />
<br />
Developers of applications that utilize open data are usually individuals or small, agile companies. When the providers of the open data are large organizations that cannot respond quickly to developers’ needs, there is going to be friction. Here are some tips for organizations that want developers to be able to smoothly leverage their open data sets:<br />
<br />
<ul>
<li>Prefer open source components and universal data formats for developing APIs. This avoids provider lock-in and makes maintenance easier.</li>
<li>Outsource carefully and have someone in your organization with sufficient technical know-how oversee the project. Facilitate knowledge transfer by bringing consultants in-house and having them collaborate with your own people instead of doing development off-site.</li>
<li>Designate someone to be the “API maintainer”. This person should coordinate development, answer questions from developers, monitor API usage and watch out for potential problems. Provide this person’s contact information to developers. </li>
<li>Open a public issue queue on for example <a href="https://github.com/">GitHub</a>. This provides a channel for developers to report issues, to discover what problems other developers have run into and and to see what progress has been made on resolving them.</li>
<li>Transparency is key - provide documentation, examples, a list of known bugs and so on. Consider open-sourcing your API. </li>
</ul>
<br />
I think the push for making data open and accessible is an applaudable effort by the public sector. There have already been many beautiful applications built using open data - take, for example, <a href="http://blindsquare.com/">BlindSquare</a> or our own <a href="http://events.hubi.fi/">Linked Events</a> aggregator. To make open data even more useful, data providers should seek to become more agile and responsive to developers’ needs. A public-facing API can’t be implemented and then forgotten about, it is something that needs to be supported and refined continuously.<br />
<br />Anonymousnoreply@blogger.com0tag:blogger.com,1999:blog-8954277266683132656.post-43125225196819168012013-09-25T09:34:00.000+03:002013-09-27T10:54:20.839+03:00Querydsl in Finnish eDemoracy and eServices projects<p>Querydsl, our open source database query library, is used in multiple business sectors to make querying databases easier and more productive. Besides various private sector projects Querydsl is used at least in two large-scale public sector projects of the <a href="http://www.vm.fi/vm/en/05_projects/03_sade/index.jsp"><b>Action Programme on eServices and eDemocracy</b></a> (SADe) of the Finnish Ministry of Finance here in Finland.</p>
<p>The main goal of the SADe project is to develop public services for individuals, companies and other entities. These customer-oriented and interoperable services are meant to increase the quality and efficiency of public sector services. The SADe programme consists of eight projects which are managed by the Ministry of Finance and other ministries:</p>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipi1vtLMmSxD10AO8IXNulaaYBRtXVL23vs49q9QHe9p7nM5goKwo3vl07PLBg2wxJ0wGXxs4YzDsQFEueE-qc39op6b9zhFLMJrg3kcJLZL4Cxb3NJ_otMnlzvHNUbUzXoOnUN8r4HhU/s1600/sade_en.vip.GIF" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipi1vtLMmSxD10AO8IXNulaaYBRtXVL23vs49q9QHe9p7nM5goKwo3vl07PLBg2wxJ0wGXxs4YzDsQFEueE-qc39op6b9zhFLMJrg3kcJLZL4Cxb3NJ_otMnlzvHNUbUzXoOnUN8r4HhU/s1600/sade_en.vip.GIF" /></a>
<p>Querydsl is used in at least two of these projects (source code will be open by the end of this year):</p>
<p>In the <a href="http://oikeusministerio.fi/fi/index/valmisteilla/kehittamishankkeita/osallistumisymparisto-projekti.html"><b>Finnish eParticipation Environment</b></a> of the Ministry of Justice, or more specifically in the <a href="https://www.kansalaisaloite.fi/fi/ohjeet/briefly-in-english"><b>Kansalaisaloite.fi</b></a> portal, Querydsl is used on top of SQL databases. We have found that handwritten SQL is usually hard to maintain, error-prone, and slow. Querydsl basically solves all of these problems.</p>
<p><a href="http://www.oph.fi/oppijanpalvelut"><b>Learners’ Online Services</b></a> of the Ministry of Education and Culture is also using Querydsl. Their use case is a bit different: they have SQL and NoSQL databases, Querydsl keeps the developers productive when switching between the two due to its consistent query API. Mysema has provided consulting to the Learners’ Online Services developers on how to get the most out of Querydsl.</p>
<p>We used Querydsl on top of <a href="http://lucene.apache.org/core/">Apache Lucene</a> when developing <a href="http://suomenmuseotonline.fi"><b>Suomen Museot Online</b></a>. This site provides a browsable database of almost 200,000 museum items. It gathers data from nearly 50 Finnish museums and makes the data available to <a href="http://www.europeana.eu/">Europeana</a>.</p>
<p>If you are interested in Mysema’s consulting services, please contact <a href="mailto:sales@mysema.com">sales@mysema.com</a>.</p>
Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com0tag:blogger.com,1999:blog-8954277266683132656.post-43520482084237853472013-09-03T11:38:00.000+03:002013-09-03T15:41:03.069+03:00Euroopan unionin tietosuojalainsäädäntö uudistumassa (This post is only available in Finnish)<br />
<br />
Kiitos Edward Snowdenin tietoturvasta ja henkilötietojen suojaamisesta on tullut viime aikoina ajankohtaisia puheenaiheita. Yhdysvaltojen PRISM-ohjelman laajuus on tietenkin tullut yllätyksenä, mutta ohjelmistojen suunnitteluun ja implementointiin meillä Suomessa se ei kuitenkaan suoraan vaikuta. Toinen, paljon merkittävämpi seikka, joka ei ainakaan tähän asti ole hirveästi julkisuutta saanut, on Euroopan unionin uusi tietosuojalainsäädäntö.<br />
<br />
Uutta tietosuoja-asetusta säädetään tällä hetkellä Euroopan unionissa ja se tulee voimaan mahdollisesti noin kahden vuoden päästä. Asetuksena se tulee jäsenvaltioissa tietyn siirtymäajan jälkeen automaattisesti voimaan, toisin kuin esimerkiksi direktiivi, jota jäsenvaltio pystyy vielä muokkaamaan ennen voimaanastumista. Suomen kohdalla uusi asetus korvaa henkilötietolain vuodelta 1999.<br />
<br />
Keskustelu uudesta tietosuojalainsäädännöstä on tähän asti kohdistunut isoihin yrityksiin, esimerkiksi Facebookiin ja Googleen, jotka prosessoivat valtavia määriä henkilötietoja. Pitää kuitenkin muistaa, että uusi lainsäädäntö tulee koskemaan kaikkia tahoja, yrityksiä ja organisaatioita, jotka jollain tavalla käsittelevät henkilötietoja. Yhtä paljon lainsäädäntö koskee Suomen valtiota, kuntia, seurakuntia ja vaikka videovuokraamoja kuin kansainvälisiä Internet-yrityksiä. Uuden asetuksen myötä laiminlyönti tietosuoja-asioissa voi johtaa isoihin sakkoihin.<br />
<br />
Uuden asetuksen noudattaminen vaatii, että henkilötietojen suojaaminen ja oikeaoppinen käyttö otetaan alusta asti huomioon ohjelmistoprojektien suunnittelussa ja toteutuksessa. Tietosuoja ei tietenkään ole ohjelmistokehitysprojekteissa mikään uusi vaatimus, mutta tekniset haasteet lisääntyvät tuntuvasti uuden asetuksen myötä. On esimerkiksi ehdotettu, että henkilöllä, jonka tietoja järjestelmässä käytetään, tulisi olla mahdollisuus kaikkien omien tietojen poistamiseen. Tietoja pitää myös pystyä viemään ulos järjestelmästä ”yleisesti käytetyssä formaatissa” helpottaakseen siirtoa yhdeltä palveluntarjoajalta toiselle. Facebookin kokoisella yrityksellä on tarpeeksi resursseja toteuttaa järjestelmissään tällaiset ominaisuudet. Pienen kunnan rakennuslautakunnalle tai vaikka musiikkikoululle uudet vaatimukset voivat kuitenkin olla hyvinkin haasteellisia.<br />
<br />
Uuden asetuksen myötä henkilötietoja käsittelevälle organisaatiolle syntyy uusia velvollisuuksia. Mahdollisista tietosuojaloukkauksista tulee esimerkiksi ilmoittaa tietosuojaviranomaiselle ja käyttäjille. Organisaation pitää pystyä näyttämään, että henkilötietoja ei tallenneta vailla käyttötarkoitusta, ja että ne poistetaan automaattisesti, kun niitä ei enää tarvita.<br />
<br />
On todennäköistä, että uuden tietosuoja-asetuksen vaatimukset - etenkin tekniset - voivat tulla yllätyksenä monelle tietojärjestelmän tilaajalle. Mysemalla seuraamme tietosuojalainsäädännön kehitystä tarkasti ja otamme tarvittaessa asian hyvissä ajoin puheeksi, jotta asiakas voi olla varma olemassa olevan lainsäädännön noudattamisesta.<br />
<br />
Euroopan unionin ehdottaman tietosuoja-asetuksen päämäärä on luoda organisaatioissa kulttuuri, jossa henkilötietojen käsittelyä ja suojaamista huomioidaan laajemmin ja perusteellisemmin kuin nykypäivänä. Organisaatiot, jotka jo hyvissä ajoin ottavat asetuksen vaatimat asiat huomioon, ovat kilpailussa etulyöntiasemassa.<br />
<div>
<br /></div>
Anonymousnoreply@blogger.com1tag:blogger.com,1999:blog-8954277266683132656.post-50374462983289117552013-08-30T13:35:00.001+03:002013-09-01T13:01:05.630+03:00clojuTRE 2013<p>We attended <a href="http://clojutre.org/2013/">clojuTRE 2013</a> this week and it was brilliant, thanks to <a href="http://metosin.fi/">Metosin</a> and <a href="http://nitor.fi/">Nitor</a> for organizing.</p>
<h2>The presentations:</h2>
<ul>
<li><a href="http://zenrobotics.fi/">ZenRobotics</a>, who can probably claim to have deployed <a href="http://clojure.org/">Clojure</a> into production for the longest here in Finland, had <a href="https://twitter.com/jks">Jouni Seppänen</a> and <a href="https://plus.google.com/104557509369826411263/about">Joel Kaasinen</a> talk about the challenges with resource handling and how to integrate with OpenCV.</li>
<li><a href="https://twitter.com/teropa">Tero Parviainen</a> gave a talk on <a href="http://pedestal.io/">Pedestal</a>, an über hipster web development framework by <a href="http://thinkrelevance.com/">Relevance</a>. Tero's opinion was that it is still evolving and probably not something you would use for customer projects just yet.</li>
<li>International Clojure superstars <a href="https://twitter.com/samaaron">Sam Aaron</a>, <a href="https://twitter.com/cgrand">Christophe Grand</a>, <a href="https://twitter.com/edmundjackson">Edmund Jackson</a> and <a href="https://twitter.com/kotarak">Meikel Brandmeyer</a> from <a href="http://www.lambdanext.com/">Lambda Next</a> gave a presentation on <a href="https://github.com/clojure/core.async">core.async</a>. The integration of a technical presentation with a comedy act was hilarious. For this topic it worked perfectly, even though there were a few deadlocks.</li>
<li>Last but not least was a weapons demonstration by <a href="https://twitter.com/jarppe">Jari Länsiö</a> from <a href="http://www.metosin.fi/">Metosin</a>. This super enthusiastic performance showed how to control an (almost) military grade drone with Clojure.</li>
</ul>
<p>All in all the presentations were excellent and easily matched those at more established conferences.</p>
<h2>After-party</h2>
<p>After the official program most of us headed to the after-party. It was very interesting to share thoughts with other Finnish Clojurians on how to convince customers that Clojure, and its ecosystem, is production ready.</p>
<h2>Final thoughts</h2>
<p>We deployed our first Clojure project over a year ago and we have luckily been successful in proving our customers that Clojure is a viable option so that the majority of our new projects are Clojure-based. We found that some of the best arguments for Clojure are development speed, clear paradigm and consistent style. The last two lessen vendor lock-in which is always a worry for the customer.</p>
Vesa Marttilahttp://www.blogger.com/profile/05764922455975275626noreply@blogger.com0tag:blogger.com,1999:blog-8954277266683132656.post-73223965746923409402013-03-28T17:42:00.000+02:002013-04-02T13:28:30.984+03:00Querydsl 3.0 in a nutshellLast week we released Querydsl 3.0. The goal of this release was to streamline Querydsl usage. In this post I will highlight a few of the general changes and some JPA/SQL specific ones.
<h3>Naming changes</h3>
Naming of query and subquery classes has been unified into the following form
Query classes:
<table>
<thead>
<tr>
<th width="20%">Module</th>
<th width="40%">before</th>
<th width="40%">after</th>
</tr>
</thead>
<tbody>
<tr>
<th>Collections</th>
<td>ColQuery (interface) - ColQueryImpl</td>
<td>CollQuery</td>
</tr>
<tr>
<th>JDO</th>
<td>JDOQLQuery (interface) - JDOQLQueryImpl</td>
<td>JDOQLQuery (interface) - JDOQuery</td>
</tr>
<tr>
<th>SQL</th>
<td>SQLQuery (interface) - SQLQueryImpl</td>
<td>SQLQuery</td>
</tr>
</tbody>
</table>
<p>No changes in JPA, Mongodb, Lucene and Hibernate Search.</p>
Subquery classes:
<table>
<thead>
<tr>
<th width="20%">Module</th>
<th width="40%">before</th>
<th width="40%">after</th>
</thead>
<tbody>
<tr>
<th>JPA</th>
<td>JPQLSubQuery - JPASubQuery - HibernateSubQuery</td>
<td>JPQLSubQuery (interface) - JPASubQuery - HibernateSubQuery</td>
</tr>
<tr>
<th>SQL</th>
<td>SQLSubQuery</td>
<td>SQLSubQuery</td>
</tr>
</tbody>
</table>
<p><a href="https://github.com/mysema/querydsl/issues/186">https://github.com/mysema/querydsl/issues/186</a></p>
<h3>Tuples as first-class citizens</h3>
We encountered the following pattern often in our code
<pre name="code" class="brush:java">
query.list(new QTuple(...))
</pre>
and decided to change the varargs list method to return a list of tuples instead of object arrays.
Thus
<pre name="code" class="brush:java">
List<Tuple> result = query.list(new QTuple(a, b, …))
</pre>
calls can be refactored into
<pre name="code" class="brush:java">
List<Tuple> result = query.list(a, b, …)
</pre>
The same change has been performed for the other varargs projection methods of the <b>Projectable</b> interface.
<p><a href="https://github.com/mysema/querydsl/issues/296">https://github.com/mysema/querydsl/issues/296</a></p>
<h3>Unified distinct usage</h3>
The following distinct usage has been deprecated
<pre name="code" class="brush:java">
query.listDistinct(a, b, c)
</pre>
The preferred usage is now
<pre name="code" class="brush:java">
query.distinct().list(a, b, c)
</pre>
Which is more in line with other modifiers that are applied to the query.
<p><a href="https://github.com/mysema/querydsl/issues/303">https://github.com/mysema/querydsl/issues/303</a></p>
<h3>Deeper defaults for path initialization</h3>
Querydsl metaclasses are now initialized one level further, so now there are less risks of running into issues when deep paths are used.
And if you want to be absolute sure that all paths are initialized you can always use the accessor option as documented <a href="http://www.querydsl.com/static/querydsl/3.1.0/reference/html/ch03s04.html#d0e1699">here</a>.
<p><a href="https://github.com/mysema/querydsl/issues/301">https://github.com/mysema/querydsl/issues/301</a></p>
<h3>Improved iteration for JPA</h3>
The <b>iterate</b> methods in Querydsl JPA query classes have been optimized to utilize Hibernate and EclipseLink specific iteration support.
For Hibernate <b>ScrollableResults</b> are used and for EclipseLink <b>Cursors</b>.
<p><a href="https://github.com/mysema/querydsl/issues/346">https://github.com/mysema/querydsl/issues/346</a></p>
<h3>Batoo JPA support</h3>
<a href="http://batoo.jp/">Batoo JPA</a> is now officially supported by Querydsl.
<p><a href="https://github.com/mysema/querydsl/issues/265">https://github.com/mysema/querydsl/issues/265</a></p>
<h3>Roo support</h3>
The <b>@RooJpaEntity</b> annotation of the <a href="http://www.springsource.org/spring-roo">Spring Roo</a> project is now officially supported via the <b>com.mysema.query.apt.roo.RooAnnotationProcessor</b> processor. The workaround of using GenericExporter is no longer necessary.
<p><a href="https://github.com/mysema/querydsl/issues/318">https://github.com/mysema/querydsl/issues/318</a></p>
<h3>with -> on</h3>
The <b>with</b> method for Querydsl JPA queries has been replaced with <b>on</b>, to promote a more consistent line of keywords in the query interfaces.
<p><a href="https://github.com/mysema/querydsl/issues/299">https://github.com/mysema/querydsl/issues/299</a></p>
<h3>Support for order by nulls first</h3>
Explicit null ordering is now supported in Querydsl SQL. The supported extensions are
<pre name="code" class="brush:java">
query.orderBy(lastName.asc().nullsFirst())
</pre>
and
<pre name="code" class="brush:java">
query.orderBy(lastName.asc().nullsLast())
</pre>
<p><a href="https://github.com/mysema/querydsl/issues/174">https://github.com/mysema/querydsl/issues/174</a></p>
<h3>Builder methods for SQLTemplates types</h3>
The previous method of customizing SQLTemplates subclasses was a little awkward and involved further subclassing like this
<pre name="code" class="brush:java">
new H2Templates(true){{
newLineToSingleSpace();
}}
</pre>
Querydsl 3 promotes now the usage of builder classes for customization
<pre name="code" class="brush:java">
H2Templates.builder()
.quote()
.newLineToSingleSpace()
.build();
</pre>
<p><a href="https://github.com/mysema/querydsl/issues/355">https://github.com/mysema/querydsl/issues/355</a></p>
<h3>Conclusion</h3>
The full list of changes for the 3.0 release are in <a href="https://groups.google.com/forum/#!topic/querydsl/LLagS2Jh2s4">3.0.0</a>,
<a href="https://groups.google.com/forum/#!searchin/querydsl/3.0.0/querydsl/T8ey0KqPWOg/RAQdZ9Hu6ToJ">3.0.0.BETA2</a>
and <a href="https://groups.google.com/forum/#!searchin/querydsl/3.0.0/querydsl/nQ27HMcgI1o/XvtDyR1063YJ">3.0.0.BETA1</a>.
<p>The latest stable release of Querydsl is 3.1. It fixes some packaging issues of the 3.0 release: <a href="https://groups.google.com/forum/#!topic/querydsl/I562uKQlheE">https://groups.google.com/forum/#!topic/querydsl/I562uKQlheE</a>. Usage of the 3.1 release is recommended.</p>
Happy querying!Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com0tag:blogger.com,1999:blog-8954277266683132656.post-87704031436240795252012-07-03T13:26:00.000+03:002013-05-08T16:24:04.691+03:00Modularizing JavaScript Code with Backbone.js and RequireJS<p>Lately we have been doing a lot of JavaScript development and have been very happy about the improvements in libraries and frameworks, long gone are the days of seeing JavaScript just as a necessary evil. Unfortunately there is still legacy code lying around even in our codebase.</p>
<p>As an example I present an application that was developed by us: <a href="http://semantic.hri.fi/">Semantic.hri.fi</a>, it provides a <a href="http://linkeddata.org/">Linked Open Data</a> view of <a href="http://www.aluesarjat.fi/">www.aluesarjat.fi</a> statistics. What we will be looking at is the <a href="http://semantic.hri.fi/sparql.html">SPARQL search page</a> and mainly its <a href="http://semantic.hri.fi/scripts/sparql.js">source code</a>.</p>
<p>Even with a quick glance it is easy to spot several issues which are nowadays handled with a different approach:</p>
<ul>
<li>String prototype is monkey patched with a function that is only used once</li>
<li>HTML is constructed in code via string concatenation</li>
<li>Application is not properly layered due to rendering happening in back-end data fetches</li>
<li>Hard to tell which functions are managing which area of the screen</li>
<li>File is very long</li>
<li>Global namespace is polluted with several variables</li>
</ul>
<p>Now that we have identified some of the problem spots, how would we fix them?</p>
<h3>String prototype is monkey patched with a function that is only used once</h3>
<p>Monkey patching causes compatibility issues and thus is not very sensible. Luckily an easy fix is just to use a function that accepts the string as an argument. Another option of course is to use an external library for handling string manipulation, such as <a href="http://epeli.github.com/underscore.string/">Underscore.string</a>.</p>
<h3>HTML is constructed in code via string concatenation</h3>
<p>Instead of manually constructing HTML like this:</p>
<pre class="brush:javascript" name="code">
//...
html.push("<thead><tr>");
for (var i = 0; i < vars.length; i++){
html.push("<th>" + vars[i] + "</th>");
}
html.push("</tr></thead>");
// ..
</pre>
<p>I would use a templating library such as <a href="http://handlebarsjs.com/">Handlebars</a>. With Handlebars we would have a template declared in its own file like this:</p>
<pre class="brush:html" name="code">
...
<thead>
<tr>
{{#each vars}}
<th>{{this}}<th>
{{/each}}
</tr>
</thead>
...
</pre>
<p>And the code to populate it with:</p>
<pre class="brush:javascript" name="code">
var template = Handlebars.compile(tableTemplate);
$('#results').html(template({vars: vars}));
</pre>
<h3>Application is not properly layered due to rendering happening in back-end data fetches</h3>
<p>Instead of doing DOM manipulation inside an AJAX success callback like this:</p>
<pre class="brush:javascript" name="code">
success: function(data){
var defaultNamespaces = [];
var bindings = data.results.bindings;
for (var i = 0; i < bindings.length; i++){
var binding = bindings[i];
namespaces[binding["prefix"].value] = binding["ns"].value;
defaultNamespaces.push("PREFIX ", binding["prefix"].value, ": <", binding["ns"].value, "> </br>\n");
}
$("#namespaces").html(defaultNamespaces.join(""));
init();
}
</pre>
<p>I would rather just fire an event to inform anyone who is interested that we are done loading. For this we use a common event aggregator called <em>vent</em>, which is just a JavaScript object with <a href="http://documentcloud.github.com/backbone/#Events">Backbone.Events</a> mixed in. With <em>vent</em> in use I would change the above to something like this:</p>
<pre class="brush:javascript" name="code">
success: function(data) {
vent.trigger('query:success', data.results.bindings);
}
</pre>
<p>This enables the <em>success</em> function to only care about retrieving the data. How the data is processed later on is left to the subscriber(s).</p>
<h3>Hard to tell which functions are managing which area of the screen</h3>
<p>Looking at the code I am having a really hard time understanding which functions are responsible for managing which areas of the screen, they should definitely be somehow bundled. A great way to do this is with <a href="">Backbone.View</a>s.</p>
<p>I would probably first create a <em>SparqlView</em> which is composed of <em>SearchView</em> (one on the left) and <em>SavedQueriesView</em>. After that I would probably split <em>SearchView</em> into two separate views <em>QueryView</em> and <em>ResultsView</em>. Below is a pseudocode example of the structure:</p>
<pre>
SparqlView
SearchView
QueryView
ResultsView
SavedQueriesView
</pre>
<p>Backbone.Views would probably be used for smaller elements as well, but these changes alone would give us a better understanding of the structure, plus the benefit of all the goodies Backbone.View provides.</p>
<h3>File is very long</h3>
<p>This can be of course fixed by splitting the file into separate files and then loading them separately. Luckily for us, we have already split the code into separate Backbone entities so moving them into their own files is really easy:</p>
<table>
<tr>
<th>View</th><th>Filename</th>
</tr>
<tr>
<td>SparqlView</td><td>js/views/sparql.js</td>
</tr>
<tr>
<td>SearchView</td><td>js/views/sparql/search.js</td>
</tr>
<tr>
<td>QueryView</td><td>js/views/sparql/search/query.js</td>
</tr>
<tr>
<td>ResultsView</td><td>js/views/sparql/search/results.js</td>
</tr>
<tr>
<td>SavedQueriesView</td><td>js/views/sparql/saved_queries.js</td>
</tr>
</table>
<p>When the subview has only one parent I find it a good practice to put the subview into a subdirectory with the name of their parent.</p>
<h3>Global namespace is polluted with several variables</h3>
<p>We have split the code into separate files, but have not really done anything about polluting the global namespace and our HTML file is full of script sources. I would bring in <a href="http://requirejs.org/">RequireJS</a> to sort out this mess.</p>
<p>With RequireJS we declare only our application entry point in the HTML:</p>
<pre class="brush:html" name="code">
<script data-main="main" src="js/libs/require/require.js"></script>
</pre>
<p><em>data-main</em> specifies which file is used as the entry point, in our case it is main.js and its contents is something like the following:</p>
<pre class="brush:javascript" name="code">
require(['app'], function(App) {
App.initialize();
});
</pre>
<p><em>require</em> is a function that takes an array as its first argument and a function as its second. The values in the array are RequireJS modules which are fed into the function as arguments. So in this case the argument <em>App</em> is module app.js. Inside the function we call <em>App</em>'s method initialize, let's have a look at app.js:</p>
<pre class="brush:javascript" name="code">
define(['js/libs/jquery/jquery', 'js/views/sparql.js'],
function($, SparqlView) {
var initialize = function() {
new SparqlView({el: $('#sparql')});
}
return {initialize: initialize};
});
</pre>
<p>Here we have defined a RequireJS module that has two dependencies: jQuery and SparqlView and we only expose a hash containing one function, <em>initialize</em>. When called, <em>initialize</em> creates a new SparqlView, SparqlView is defined as a RequireJS module as all of its subviews.</p>
<p>We now hopefully have a clear view on how to take old JavaScript code to the present.</p>Vesa Marttilahttp://www.blogger.com/profile/05764922455975275626noreply@blogger.com1tag:blogger.com,1999:blog-8954277266683132656.post-11320672548651870112012-06-21T15:29:00.000+03:002013-05-08T16:24:30.270+03:00Form Data Extraction with Backbone.js, Underscore.js and jQuery<p>Our weapon of choice for building single-page web applications is <a href="http://documentcloud.github.com/backbone/">Backbone.js</a>. Unlike a few of its competitors, such as <a href="http://emberjs.com/">Ember.js</a>, <a href="http://knockoutjs.com/">Knockout</a>, Backbone does not support model binding out of the box.
There are a some libraries that add model binding support to Backbone, probably first on the block was <a href="https://github.com/derickbailey/backbone.modelbinding">Backbone.ModelBinding</a> which has now been discontinued, <a href="https://github.com/theironcook/Backbone.ModelBinder">Backbone.ModelBinder</a> and <a href="https://github.com/amccloud/backbone-bindings">Backbone Bindings</a>.</p>
<p>Even though these libraries make sense in some cases, we have noticed that usually when starting out, Backbone's default tools (Backbone itself, <a href="http://documentcloud.github.com/underscore/">Underscore.js</a> and <a href="http://jquery.com/">jQuery</a>) are enough to get the ball rolling, as demonstrated below.</p>
<h3>HTML</h3>
<pre class="brush:html" name="code">
<form>
Name: <input type="text" name="name"/>
Age: <input type="text" name="age"/>
<input type="submit" value="Submit"></input>
</form>
</pre>
<h3>JavaScript</h3>
<pre class="brush:javascript" name="code">
var UserForm = Backbone.View.extend({
events: {'submit': 'save'},
initialize: function() {
_.bindAll(this, 'save');
},
save: function() {
var arr = this.$el.serializeArray();
var data = _(arr).reduce(function(acc, field) {
acc[field.name] = field.value;
return acc;
}, {});
this.model.save(data);
return false;
}
});
var userForm = new UserForm({el: this.$('form'), model: new User()});
</pre>
<p>To understand the actual data extraction I will explain <a href="http://api.jquery.com/serializeArray/">serializeArray</a> and <a href="http://documentcloud.github.com/underscore/#reduce">reduce</a>.</p>
<p><strong>serializeArray</strong> is a jQuery method that takes a form and constructs an array of all the input elements that have a <em>name</em> attribute. The resulting array is in the following form:</p>
<pre class="brush:javascript">[{name: "name", value: "John Smith"}, ...]</pre>
<p>An array of maps is not quite something we are looking for, so we have to do another transformation.</p>
<p><strong>reduce</strong> is used to build up a new value from the array with an original seed value (empty map) and a function. The function's first argument <em>acc</em> is the return value of the previous function call and the second argument <em>field</em> is the current element in the array. Now we have something that can be passed to <a href="http://documentcloud.github.com/backbone/#Model-save">Backbone.Model.save</a>:</p>
<pre class="brush:javascript">{name: "John Smith", age: 34}</pre>Vesa Marttilahttp://www.blogger.com/profile/05764922455975275626noreply@blogger.com4tag:blogger.com,1999:blog-8954277266683132656.post-19952595838396582632012-03-09T12:41:00.012+02:002012-03-11T14:47:19.785+02:00Scalagen - Java to Scala Conversion<p><a href="https://github.com/mysema/scalagen">Scalagen</a> was born out of the idea that we would like to port some of our projects from Java to Scala source code. Instead of porting all the code manually a tool could be used to do most of the bulk work. We did some initial experiments with <a href="http://code.google.com/p/jatran/">Jatran</a>, but found it lacking.</p><p>After that we investigated some options to write such a tool ourselves. First we needed a parser for Java sources to turn into abstract syntax trees. We decided to use the <a href="http://code.google.com/p/javaparser/">Javaparser</a> framework since it supports Java 5 source and had an easy to use API.</p><h3>Initial Architecture</h3><p>Then work began on Scalagen. The initial parts were written in Java, such as a visitor implementation for the javaparser which prints out the tree as Scala syntax. After that had been done this visitor was used to convert the visitor class from Java to Scala. After that the implementation of Scalagen continued to be fully Scala-based.</p><p>After that we wrote visitor implementations which did transformations on the AST. Needed transformations were to extract static members into companion objects and to change various control statements into a more Scala-esque form. At first the visitor would just mutate the original AST from one state to another. This proved to cause problems with Scala's immutable collection types so we rewrote the transforming visitors to always return new detached AST versions.</p><p>Now the initial architecture was set up and further transformations could be written. After a while we had an impressive list of transformations which could be easily used via both their class and object variants:</p><pre class="brush:scala">object Primitives extends Primitives
class Primitives extends UnitTransformerBase {
...
}
</pre><h3>Wrapping the Parser</h3><p>Then we began to adapt the javaparser API to be more easy to use. We declared short type aliases in a singleton object to strip off redundant suffixes such as Stmt and Expr:</p><pre class="brush:scala">type Annotation = AnnotationExpr
type AnnotationDecl = AnnotationDeclaration
type AnnotationMember = AnnotationMemberDeclaration
type Assign = AssignExpr
type Binary = BinaryExpr
type Block = BlockStmt
type BodyDecl = BodyDeclaration
</pre><p>Then we began to write deconstructors to make AST pattern matching more concise:</p><pre class="brush:scala">object FieldAccess {
def unapply(f: FieldAccess) = Some(f.getScope, f.getField)
}
object For {
def unapply(f: For) = Some(toScalaList(f.getInit), f.getCompare,
toScalaList(f.getUpdate), extract(f.getBody))
}
object Foreach {
def unapply(f: Foreach) = Some(f.getVariable, f.getIterable,
extract(f.getBody))
}
object MethodCall {
def unapply(m: MethodCall) = Some(m.getScope, m.getName,
toScalaList(m.getArgs))
}
</pre><p>And something more complex which provides both a new-less constructor via apply and a deconstructor via unapply.</p><pre class="brush:scala">object VariableDeclaration {
def apply(mod: Int, name: String, t: Type): VariableDeclaration = {
val variable = new VariableDeclarator(new VariableDeclaratorId(name))
new VariableDeclaration(mod, t, variable :: Nil)
}
def unapply(v: VariableDeclaration) =
Some(v.getType, toScalaList(v.getVars))
}
</pre><p>The singleton names matched the type aliases so we had a very Scala-like meta-layer on top of the javaparser AST classes.</p><p>Here is a fairly complex example of how the pattern matching could be used in the transforming visitors.</p><pre class="brush:scala">override def visit(nn: Foreach, arg: CompilationUnit): Node = {
val n = super.visit(nn, arg).asInstanceOf[Foreach]
n match {
case Foreach(
VariableDeclaration(t, v :: Nil),
MethodCall(scope, "entrySet", Nil), body) => {
val vid = v.getId.toString
new Foreach(
VariableDeclaration(0, "(key, value)", Type.Object),
scope, n.getBody.accept(toKeyAndValue, vid).asInstanceOf[Block])
}
case _ => n
}
}
</pre><p>This looks still quite cryptic if you are not familiar with the AST structure of javaparser, but for the ones familiar with structure of the AST, this is a fairly intuitive way to match AST patterns.</p><h3>Packaging</h3><p>Scalagen provides direct Maven support via a plugin. You can use it directly via the command line like this</p><pre class="brush:java">mvn com.mysema.scalagen:scalagen-maven-plugin:0.1.3:main \
-DtargetFolder=target/scala
</pre><p>and for test sources</p><pre class="brush:java">mvn com.mysema.scalagen:scalagen-maven-plugin:0.1.3:test \
-DtargetFolder=target/scala
</pre><p>Here is the snippet for an explicit configuration in a POM:</p><pre class="brush:xml"><plugin>
<groupId>com.mysema.scalagen</groupId>
<artifactId>scalagen-maven-plugin</artifactId>
<version>0.1.3</version>
</plugin>
</pre><p>To convert main sources run</p><pre class="brush:java">mvn scalagen:main
</pre><p>and to convert test sources run</p><pre class="brush:java">mvn scalagen:test
</pre><p>The conversion results are to be seen as a starting point for the Java to Scala conversion. Some elements are not transformed correctly for various reasons and will need manual intervention.</p><h3>Finally</h3><p>Scalagen is an experimental effort and has still lots of rough edges, but we are open to improvement suggestions and stylistic changes.</p>Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com4tag:blogger.com,1999:blog-8954277266683132656.post-73509934892740943992011-11-01T17:13:00.000+02:002011-11-02T23:02:08.620+02:00On Ontology creationLast week I held a presentation at the Kirjastoverkkopäivät event in Helsinki and talked about the stages of Ontology creation. Instead of posting the slides here I decided to elaborate more on the topics of the presentation.<br />
The need for ontology creation arises often from having to agree on a formal terminology. Formal terminologies are important to share understanding and to reduce ambiguity. And to create a formal terminology a conceptual framework is needed to build on. This is where ontologies come in.<br />
An ontology is a formal representation of knowledge as a set of concepts within a domain, and the relationships between those concepts.<br />
<h3>
Ontology standards</h3>
To be able to share ontologies and to reuse other ontologies in fragments or full form a common agreement on the building blocks of ontologies and their meaning is needed. The <a href="http://www.w3.org/standards/semanticweb/">Semantic Web</a> effort of the World Web Foundation has been dealing with this goal and has published several standards to tackle the problem.<br />
The most relevant ontology standards today are <a href="http://www.w3.org/TR/rdf-schema/">RDF Schema</a>, OWL <a href="http://www.w3.org/TR/owl-features/">1.0</a> & <a href="http://www.w3.org/TR/owl2-overview/">2.0</a> and <a href="http://www.w3.org/2009/08/skos-reference/skos.html">SKOS</a>.<br />
<ul>
<li>
<b>RDF Schema example</b><br/>
published in 1998, standardized in 2004
</li>
<li>
<b>OWL example</b>
<ul>
<li>1.0 (2004)<br/>
divided into the dialects Light, DL (Description Logic) and Full, extends RDF Schema
</li>
<li>2.0 (2009)<br/>
divided into the profiles EL, QL and RL
</li>
</ul>
</li>
<li>
<b>SKOS example</b> <br/>
ontology for vocabularies and thesauri<br />
</li>
</ul>
RDF Schema and OWL are ontology frameworks which provide a type system of classes, properties and instances and SKOS can be seen as a ontology which defines concept types and relations between concepts. <br />
The following sections will show examples to illustrate the differences and similarities of the presented standards.<br />
<h4>RDF Schema</h4>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWhu7e83Odv0dYr1mKnciBNLpKAwp6arYOVw_BSXrmTnhj6tLj-_RcPmvclZMgUz1yn6edktNJ66JOlU6hqpt7New8DaH3ylfqcXXlho3gnHgx9h2o1kh1t200szqvgTwz20k_TuXGGrM/s1600/wine.png" imageanchor="1" style="float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="255" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWhu7e83Odv0dYr1mKnciBNLpKAwp6arYOVw_BSXrmTnhj6tLj-_RcPmvclZMgUz1yn6edktNJ66JOlU6hqpt7New8DaH3ylfqcXXlho3gnHgx9h2o1kh1t200szqvgTwz20k_TuXGGrM/s320/wine.png" width="320" /></a>
The <a href="http://www.w3.org/TR/2003/PR-owl-guide-20031215/wine">Wine ontology</a> is a popular RDF Schema example as it presents in compact form a lot of RDF schema building blocks. The boxes of the images present classes, predicates and instances while the lines present properties. The ElyseZinfandel box for example is connected to the Zinfandel box via the rdf:type relation which means the resource ElyseZinfandel has the type Zinfandel. It is also connected to the Elyse box, but via the hasMaker relation, and to the Red box, via the hasColor relation.<br />
The predicates <b>rdfs:subClassOf</b> and <b>rdfs:subPropertyOf</b> are used to declare subclass and subproperty relations. <b>rdfs:domain</b> declares the subject range of a property, the type this predicate applies to while <b>rdfs:range</b> declares the object range of a property, i.e. the value range of the property.<br />
<h4>OWL</h4>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhL05qt3zCtAQVEBf5w6EFHdt3PJviAloM8kdFMoTL2DxtMYWo1rQabgoaQuX47OCjj_ll5FJch_MCqc6x3W02rBdF_8Ap8e3qJ0MEtBTDEpHmBofgViiXPsOQBSDEsmIEYcb_-xyU3ntU/s1600/bbc2.png" imageanchor="1" style="float: right; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="75" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhL05qt3zCtAQVEBf5w6EFHdt3PJviAloM8kdFMoTL2DxtMYWo1rQabgoaQuX47OCjj_ll5FJch_MCqc6x3W02rBdF_8Ap8e3qJ0MEtBTDEpHmBofgViiXPsOQBSDEsmIEYcb_-xyU3ntU/s320/bbc2.png" width="320" /></a>
The <a href="http://www.bbc.co.uk/ontologies/programmes/2009-09-07.shtml">BBC Programmes ontology</a> defines various concepts of the Media domain. This excerpt highlights the the <b>owl:disjointWith</b> relation of the OWL format.The declaration ProgrammeItem owl:disjointWith Brand for example declares that ProgrammeItem instances cannot be Brand instances and vice versa. Both ProgrammeItem and Brand are sub classes of the Programme class.<br />
<h4>
SKOS</h4>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikgJ5ITb6FEEg5RGrcf_CQcsUvXC18E5ft_pTuqkgYks_ejcBJS2AvDYW4NcU2vL-z6lJkh8cbj7HbpyreLvTsObJKyOGhrwmnaz8gSgyQlCcxIOCwRp49GpDYL6cbUdbYjlajmAVdNn8/s1600/skos2.png" imageanchor="1" style="float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikgJ5ITb6FEEg5RGrcf_CQcsUvXC18E5ft_pTuqkgYks_ejcBJS2AvDYW4NcU2vL-z6lJkh8cbj7HbpyreLvTsObJKyOGhrwmnaz8gSgyQlCcxIOCwRp49GpDYL6cbUdbYjlajmAVdNn8/s320/skos2.png" width="235" /></a>
The magazine ontology of ONKI defines concepts to be used for categorization of magazines. For example Living has the Antiques subcategory while Antiques has the subcategory Antique stores. The skos:broader relation is comparable with the rdfs:subClassOf relation, but while rdfs:subClassOf is transitive, skos:broader isn't. For transitive relations the skos:broaderTransitive relation needs to be used.<br />
For a more detailed presentation of the W3C Ontology and Vocabulary formats visit <a href="http://www.w3.org/standards/semanticweb/ontology">here</a>.<br />
<h3>
Creation of your own ontology</h3>
Sometimes the available ontologies don't cover your problem domain or define semantics that are not compatible with your needs. Defining your own ontology is then a valid option.<br />
Ontology creation begins of with informal sketching of the concepts of the problem domain. Paper and pen, mind maps or any other sketching tool can be used for this. After that the main modeling dimensions should be defined. Do the concepts define a type or categorization system? Are subclass or subconcept relations the dominating relations or maybe temporal ones or something else? The answers to these questions will guide you to the selection of the appropriate modeling standard. As mentioned before RDF Schema and OWL should be picked to model type systems and SKOS for categorizations.<br />
<h4>
Relation of your own ontology to others</h4>
Sometimes most of your problem domain has already been formalized appropriately in other ontologies. In such a case you can link your own ontology to these external ones. Your own classes, predicates and concepts can be specializations and generalizations of appropriate external entities. Another common way is to define equivalence between entities via the the <b>owl:sameAs</b> relation.<br />
The owl:sameAs relation is used much in the Linked Open Data movement to connect heterogenous datasets.<br />
<h4>Where to go next?</h4>
<p>Maintenance, further development and distribution need to be taken into account as well. Having the facilities in house to tackle these aspects is a great benefit. Otherwise probably a public host for ontologies such as ONKI should be picked.</p>
<h3>
ONKI</h3>
<a href="http://onki.fi/">ONKI</a> is a central ontology host of the finnish Semantic Web scene and is maintained by the <a href="http://www.seco.tkk.fi/">SeCo</a> team of the Aalto University. ONKI hosts 72 ontologies and vocabularies altogether: 47 OWL ontologies, 2 RDF Schema ontologies and 22 SKOS vocabularies. Most of these are freely available with a liberal license. The SeCo provides support for the ONKI service via email and notifices via Twitter of service changes and problems.<br />
<h4>
Using an ONKI ontology</h4>
ONKI provides several tools to use ontologies :<br />
<ul>
<li>ONKI Browser - a faceted browser to explore ontology content
</li>
<li>SOAP interface - for service integration
</li>
<li>ONKI Selector widget - a JavaScript widget for web application integration
</li>
<li>Downloads - in RDF/XML form to use the data in your own applications
</li>
</ul>
<h4>
ONKI Browser</h4>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgz9uQhttJJseHWqMyylrIdAMuAjMlGLNbOZoWjiidGpSYa8kZVHWs1g2vKPTT2yoAn9P5WPt0KC19ON_Myfa2emb3QrxT-AV6mFDztz1rcdlSZPpIeWP_QF6-TG-627Vrv6L0os0sinYk/s1600/onki.png" imageanchor="1" style="float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="178" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgz9uQhttJJseHWqMyylrIdAMuAjMlGLNbOZoWjiidGpSYa8kZVHWs1g2vKPTT2yoAn9P5WPt0KC19ON_Myfa2emb3QrxT-AV6mFDztz1rcdlSZPpIeWP_QF6-TG-627Vrv6L0os0sinYk/s320/onki.png" width="320" /></a>
This screenshot shows a part of the <a href="http://onki.fi/fi/browser/">ONKI Browser</a> tool, a faceted search for the ONKI content. The ONKI Browser provides free text search for ontologies and concepts and provides a navigatable tree visualization of the concept inheritance.<br />
<h4>
SOAP interface</h4>
The following methods are available in the <a href="http://onki.fi/api/v1/soap/">ONKI SOAP</a> interface:<br />
<ul>
<li>getAvailableLanguages - languages used in concept names in an ontology.</li>
<li>getAvailableTypeUris - supported concept type URIs of an ontology.</li>
<li>search - searching for ontological concepts.</li>
<li>getLabel - fetching a label for the concept with a given URI.</li>
<li>expandQuery - expanding concept(s) with given URI(s) (e.g. to subconcepts) for queries.</li>
<li>getProperties - fetching properties for the concept with a given URI.</li>
<li>getConceptTree - fetching concept hierarchy for the concept with a given URI.</li>
</ul>
A REST/JSON interface is in beta phase, but the interface is still subject to change.<br />
<h4>
ONKI Selector wdiget</h4>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiewuUndejxRS9P-fk7DdM8_i4xDdRCkgsKeCylIshksMaTNKkkDwFWwXWQXW9GmTNSUvqPlt2B-Lc3PQGY7t5AB-6QN53H74PZDbfnKZBZCVdrmTce6CJHUMWZal1M9Ych6ieOTJrHUME/s1600/onki-selector.png" imageanchor="1" style="float: right; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="183" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiewuUndejxRS9P-fk7DdM8_i4xDdRCkgsKeCylIshksMaTNKkkDwFWwXWQXW9GmTNSUvqPlt2B-Lc3PQGY7t5AB-6QN53H74PZDbfnKZBZCVdrmTce6CJHUMWZal1M9Ych6ieOTJrHUME/s320/onki-selector.png" width="320" /></a>
The <a href="http://onki.fi/widget/selector/?l=en">ONKI Selector widget</a> is a JavaScript script which can easily be embedded into modern web applications. It provides autocomplete based selection of ontology concepts. It is a fairly easy integration, but provides only label and URI of the selected concept, super concepts need to be fetched separately.<br />
The displayed form fields offer the selection of a specific ontology to be targeted in the search or all, the search query and the language of the labels.<br />
<h4>
Using an ONKI ontology in your own service</h4>
While ONKI is a general ontology host, most of the ontologies are connected to the central YSO ontology, a general finnish language ontology. The following picture shows the umbrella function of the YSO ontology:<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibKDo5tC-d-rXf6BdJ6wwnJYIrAryGXeTkG9O_awon-oWMnZliiyqvRNK7AA7bZSY9v4WBpWrQFSibPdWTmKlNJ3FB6FxR68fXfey-QOEswX-LAMJH4jRcVIKXJwTXJmrNQZyBWEMPVhE/s1600/yso6.gif" imageanchor="1" style="float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="208" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibKDo5tC-d-rXf6BdJ6wwnJYIrAryGXeTkG9O_awon-oWMnZliiyqvRNK7AA7bZSY9v4WBpWrQFSibPdWTmKlNJ3FB6FxR68fXfey-QOEswX-LAMJH4jRcVIKXJwTXJmrNQZyBWEMPVhE/s320/yso6.gif" width="320" /></a>
The benefit of this approach is that when using an ONKI ontology you have a rich foundation to start with, the downside is that the amount of concepts is huge, the YSO ontology alone hosts 20.000 concepts. Avoiding clashes and semantic mismatches can be difficult with so many concepts to be taken into account.<br />
As the background of YSO is YSA, a central Finnish language glossary, it should probably be treated more as a vocabulary than ontology.<br />
<h3>
Conclusion</h3>
<p>
This article presented an overview of the ontology standards RDF Schema, OWL and SKOS with examples and a small introduction into the toolset of the ONKI service. If you have experiences with the ONKI service you'd like to share please comment on this post.</p>Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com0tag:blogger.com,1999:blog-8954277266683132656.post-82169709721442201772011-08-26T15:06:00.011+03:002012-03-09T17:56:44.882+02:00Accessing Helsinki region statistics from your webapp<p>In this post I'd like to describe how the statistical data from the <a href="http://semantic.hri.fi/">Semantic.hfi.fi</a> service can be accessed from your web application via the demo app <a href="http://www.mysema.com/areastats/">Areastats</a>. Semantic.hri.fi offers several JSON/P web services which can directly be used from any website.</p><div class="separator" style="clear: both; text-align: center;"><a href="http://www.mysema.com/areastats" imageanchor="1" style="clear:right; float:right;margin-left:1em; margin-bottom:1em"><img border="0" height="320" width="264" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJP6OeAzrHEJPx0T4uUtF7DsqmEkYiwtQEfaySNaz8XhQ0_JRTN0jyJSjck9aPJJWB86VoRAUDxmKZ-q_A7gSNEJRIEhKQSUvDdVkEZPL32GP0fTYaVat6dZO9IPXFf4nKgTK2LNDGQI8/s320/Screenshot.png" /></a></div><p>Areastats displays statistics when accessed from within the Finnish capital region. It uses internally the <br />
<ul><li>HTML 5 Geolocation API to obtain the access location as WGS 85 coordinates, </li>
<li>a simple webservice to convert the coordinates into a code for the area the location resides in, </li>
<li>the Semantic.hri.fi Faceted Search API to access statistics for this area and </li>
<li>the jQuery API for DOM manipulation and Ajax</li>
</ul></p><p>The <a href="http://code.google.com/p/geo-location-javascript/">geo-location-js</a> API is used as an adapter for the incompatible Geolocation APIs used in various browsers. It provides a convenient access point to register callbacks for action when the location could be obtained and when not:</p><pre name="code" class="brush:javascript">if (geo_position_js.init()) {
geo_position_js.getCurrentPosition(geo_success, geo_error);
}
</pre><p>The geo_success callback then retrieves the area code from a web service via Ajax and uses the fetch_stats function to process the result:</p><pre name="code" class="brush:javascript">function geo_success(p) {
$.ajax({
url: "polygon",
data: { "latitude": p.coords.latitude, "longitude": p.coords.longitude},
dataType: "json",
success: fetch_stats,
error: geo_error
});
}
</pre><p>The fetch_stats then makes an Ajax call to obtain the results: </p><pre name="code" class="brush:javascript">function fetch_stats(d) {
var alue = "alue:_" + d.KUNTA + "_" + d.PIEN + "_" + d.Nimi;
$.ajax({
url: "http://semantic.hri.fi/search",
data: {
"include": ["items"], "limit":50, "offset":0,
"value": [
alue,
"vuosi:_2010",
"yksikkö:Henkilöä",
"dataset:A025_HKI_Vakiluku1962"] },
dataType: "jsonp",
success: fetch_stats_success,
error: fetch_stats_error
});
$("#location").html(d.Nimi);
}
</pre><p>The query defines that we are interested in statistical items from the year 2010, with the unit persons (Henkilöä) and from the dataset dataset:A025_HKI_Vakiluku1962. For details on how to use the service of Semantic.hri.fi, please read the embedded <a href="http://semantic.hri.fi/">documentation</a>.</p><p>The fetch_stats_success callback then has the final results and draws a pie chart based on them: </p><pre name="code" class="brush:javascript">function fetch_stats_success(d) {
var age_index = -1;
for (var i = 0; i < d.headers.length; i++) {
if (d.headers[i] == "dimension:Ikäryhmä") {
age_index = i;
}
}
for (var i = 0; i < d.items.length; i++) {
var key = d.items[i].values[age_index];
key = key.substring(9);
$("#"+key).html(d.items[i].value);
}
drawpie();
$("#stats").show();
}
</pre>
<p>The fetch_stats_success function iterates the headers of the result to find the index of the column we are interested in and the iterates over the result items and maps the column data into the DOM of the local HTML document. After that the drawpie function is invoked and the stats can be shown.</p><p>This demo is related to the <a href="http://www.hri.fi/fi/">Helsinki Region Infoshare</a> project, which aims to provide data of the Finnish capital region in Linked Open Data form.</p><p>Feel free to contact us if you need more info on the demo or the Semantic.hri.fi services.</p>Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com1tag:blogger.com,1999:blog-8954277266683132656.post-87257869391426833992011-08-12T16:21:00.005+03:002013-05-08T16:25:00.774+03:00Querydsl in the wild<p>From time to time we observe how the Querydsl framework is used outside of Mysema to find out how Querydsl meets the needs of our user base. In this blog post I'd like to share some Querydsl usage patterns we found.</p><h3>Expression factory class</h3><p>As an alternative to Expression construction based on the core primitives reocurring constraints can be bundled as factory methods grouped by the used root types.</p><p>This example shows Expression factory methods for the User domain type</p><pre name="code" class="brush:java">package org.springframework.data.jpa.example.repository;
import org.joda.time.LocalDate;
import org.springframework.data.jpa.example.domain.QUser;
import com.mysema.query.types.expr.BooleanExpression;
class UserExpressions {
public static BooleanExpression goldMember() {
return adult().and(createdAtleastOneYearAgo());
}
public static BooleanExpression adult() {
return QUser.user.age.goe(18);
}
public static BooleanExpression createdAtleastOneYearAgo() {
return QUser.user.creationDate.loe(new LocalDate().minusYears(1));
}
public static BooleanExpression differentFrontThanLastName() {
return QUser.user.firstname.eq(QUser.user.lastname).not();
}
public static BooleanExpression usernameLongerThanSix() {
return QUser.user.username.length().gt(6);
}
}
</pre><p><a href="https://github.com/42BV/spring-data-jpa-examples/blob/570de2d2d53fd4ff3df89a60dc6c2b523e06a363/spring-data-jpa-example/src/main/java/org/springframework/data/jpa/example/repository/UserExpressions.java">Source: GitHub</a></p><h3>$ as variable name for default entity path</h3><p>In cases where Querydsl is used in combination with the Repository pattern it can be convenient to use a general variable name for the default entity path to be used. The dollar-sign is a valid candidate for this:</p><pre name="code" class="brush:java">package reevent.dao;
import com.mysema.query.jpa.JPQLQuery;
import org.springframework.stereotype.Repository;
import org.springframework.transaction.annotation.Transactional;
import reevent.domain.QUser;
import reevent.domain.User;
@Transactional
@Repository
public class UserDaoJpa extends EntityDaoBase<User> implements UserDao {
private static final QUser $ = QUser.user;
@Override
public User findByUsername(String username) {
return queryByUsername(username).uniqueResult($);
}
@Override
public boolean usernameExists(String username) {
return queryByUsername(username).exists();
}
private JPQLQuery queryByUsername(String username) {
return query().from($).where($.username.eq(username));
}
}
</pre><p><a href="https://github.com/tu-wien-ase-ss2011-qse-6/ReEvent/blob/master/src/main/java/reevent/dao/UserDaoJpa.java">Source: GitHub</a></p><p>Drawbacks with this approach are that some code inspection tools might complain about the unconventional variable name usage and that the dollar-sign is also used by the Querydsl Alias-functionality.</p><h3>Hamcrest integration</h3><p>Querydsl expressions can also be used for other tasks beside querying. A fine example is Hamcrest integration:</p><pre name="code" class="brush:java">package org.hamcrest;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.PathMatcher.valueOf;
import static org.junit.Assert.assertThat;
import org.hamcrest.domain.Address;
import org.hamcrest.domain.Customer;
import org.hamcrest.domain.QCustomer;
import org.junit.Test;
public class PathMatcherTest {
@Test
public void testPathEvaluate() {
Customer customer = new Customer();
Address address = new Address();
address.setCity("bla");
address.setStreet("blaway 142");
customer.setAddress(address);
assertThat(customer,
valueOf(QCustomer.customer.address.street,
equalTo("blaway 142")));
}
}
</pre><p>Hamcrest matchers can be used in unit tests for complex assertion construction.</p><p><a href="https://github.com/jeroenvs">Jeroen van Schagen</a>, the author of the Hamcrest example has bundled his Querydsl extensions into the <a href="https://github.com/jeroenvs/querydsl-support">querydsl-support module</a>.</p><h3>Supplying filter criteria to general find methods</h3><p>This pattern is extensively used in the Spring Data repositories. Here is a simple example:</p><pre name="code" class="brush:java">QCustomer customer = QCustomer.customer;
LocalDate today = new LocalDate();
BooleanExpression customerHasBirthday = customer.birthday.eq(today);
BooleanExpression isLongTermCustomer =
customer.createdAt.lt(today.minusYears(2));
customerRepository.findAll(customerHasBirthday.and(isLongTermCustomer));
</pre><p><a href="http://blog.springsource.com/2011/04/26/advanced-spring-data-jpa-specifications-and-querydsl/">Advanced Spring Data JPA – Specifications and Querydsl</a></p><p>If you have innovative Querydsl usage yourself, please share it with us!</p>Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com1tag:blogger.com,1999:blog-8954277266683132656.post-55177122424176573152011-03-16T10:40:00.000+02:002011-03-16T10:40:15.113+02:00Julkinen data seminaari - To 7.4.2011, Helsinki<p>Hallitus on tehnyt periaatepäätöksen (3.3.2011) julkishallinnon digitaalisten tietoaineistojen saatavuudesta. Tietoaineistojen tulee olla avoimesti saatavilla ja uudelleenkäytettävissä yhtenäisin, selkein ja kaikille tasapuolisin ehdoin. Periaatepäätös koskee ensisijaisesti sellaisenaan luovutettavissa olevia julkisia aineistoja, joiden käsittelyä ei lainsäädäntö rajoita.</p><p>Mutta mitä se tarkoittaa käytännössä? Miten julkisia tietoaineistoja tarjotaan?</p><p>Julkinen data seminaari torstaina 7.4.2011 Aalto yliopiston Chydenina-talossa Helsingissä tarjoaa tietoa julkisten tietovarantojen avaamiseen liittyen eri näkökulmilta. Seminaari on tarkoitettu kaikille tiedon parissa työskenteleville asiantuntijoille ja päättäjille kuntien ja valtion tehtävissä.</p><p>Tilaisuudessa esitellään:<br />
<ul><li>mitä Julkinen data on</li>
<li>miten valtion periaatepäätöksellä pyritään edistämään julkisen hallinnon tietovarantojen saatavuutta ja käyttöä.</li>
<li>konkreettisia esimerkkejä mitä avoimilla tietovarantojen avulla on saatu aikaiseksi meillä ja maailmalla.</li>
<li>Case Helsingin kaupungin aluesarjat-tilastotietojen avaaminen ja teknisen ratkaisun rakentaminen</li>
</ul></p><p>Tilaisuuden ohjelma:<br />
<br />
8:30 Rekisteröityminen ja aamukahvit<br />
<br />
9:00 Avaus - missä mennään julkisten tietovarantojen avauksessa, Taru Rastas, Liikenne ja Viestintäministeriö<br />
<br />
9:15 Julkinen data - mitä kaikkea se onkaan? Tapio Nurminen, FloApps Oy<br />
</p><p>10:00 Kahvitauko<br />
<br />
10:15 Case esittely - Aluesarjat-tilastot avoimeksi, Pekka Vuori, Yliaktuaari, Helsingin Kaupungin Tietokeskus<br />
<br />
10:45 Q&A + Aluesarjat Demot by FloApps & Mysema<br />
</p><p>11.30 Lounas<br />
<br />
12.30 Julkista dataa hyödyntäviä esimerkkisovelluksia ja -palveluita Suomessa ja maailmalla. Tapio Nurminen, Toimitusjohtaja, FloApps Oy<br />
<br />
13:00 Datan avaaminen käytännössä-teknisen ratkaisun rakentaminen, Samppa Saarela, Mysema Oy<br />
<br />
13:30 Avoin tieto laadukkaaksi data managementin avulla, Nino Ilveskero, Talent Base Oy<br />
</p><p>14:00 Kahvitauko<br />
<br />
14:15 Liiketoimintamallit julkisen datan avulla, Juho Lindman, Aalto Econ. CKIR<br />
<br />
14:45-15:30 Yhteenveto + kysymykset - Tapio Nurminen ja Samppa Saarela<br />
</p><p>Tilaisuuden hinta on 75 € + alv. 23%<br />
<br />
Ilmoittaudu ystävällisesti tilaisuuteen 25.3. mennessä kahvi- ja lounasjärjestelyjen johdosta ennakkoon osoitteessa:<br />
<br />
<a href="http://bit.ly/h9Hk0K">http://bit.ly/h9Hk0K</a><br />
</p><p>Seminaari järjestetään: torstaina 7. huhtikuuta, klo 8.30 - 15.30 Aalto-Yliopiston Chydenia-talossa osoitteessa Runeberginkatu 22-24, Helsinki.<br />
</p><p>Järjestäjät ovat <a href="http://www.floapps.com">Flo Apps Oy</a> ja <a href="http://www.mysema.com">Mysema Oy</a>, <br />
yhteistyössä Helsinki Regional Infoshare, Aalto Yliopiston Econ. osaston ja Talent Base Oy:n kanssa.<br />
</p>Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com0tag:blogger.com,1999:blog-8954277266683132656.post-79513885132731659472011-01-21T11:18:00.006+02:002012-03-09T18:06:48.069+02:00Querying in SQL with Querydsl<a href="http://www.querydsl.com">Querydsl</a> provides a typesafe querying layer on top of JPA, JDO, JDBC and other backends. This blog post presents a simple tutorial on how to get started with querying in SQL using Querydsl.<br />
<br />
Querydsl for SQL provides a typesafe view of the relational schema and makes expressing database queries in Java as intuitive as possible.<br />
<br />
<h3>Getting started</h3><br />
To get started with Querydsl for SQL using a Maven 2 based build environment, follow these steps.<br />
<br />
Add the following dependencies to your Maven project and make sure that the <a href="http://source.mysema.com/maven2/releases">Maven 2 repo of Mysema Source</a> is accessible from your POM :<br />
<br />
<pre name="code" class="brush:xml"><dependency>
<groupId>com.mysema.querydsl</groupId>
<artifactId>querydsl-sql</artifactId>
<version>${querydsl.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.6.1</version>
</dependency>
</pre><br />
The snippet to include the Mysema Source Maven repository is :<br />
<br />
<pre name="code" class="brush:xml"><repository>
<id>msource</id>
<url>http://source.mysema.com/maven2/releases/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</pre><br />
Set the variable querydsl.version to the latest Querydsl version available.<br />
<br />
If you are not using Maven you can download the latest builds from <a href="http://source.mysema.com/static/querydsl/latest/">here</a><br />
<br />
<h3>Creating the Querydsl query types</h3><br />
To get started with creating Querydsl SQL queries you need to create Java classes for your relational schema. This can be done either in Java, via Maven or via Ant.<br />
<br />
<h4>In Java</h4><br />
To get started export your schema into Querydsl query types like this :<br />
<br />
<pre name="code" class="brush:java">java.sql.Connection conn = ...;
MetaDataExporter exporter = new MetaDataExporter();
exporter.setPackageName("com.myproject.mydomain");
exporter.setTargetFolder(new File("src/main/java"));
exporter.export(conn.getMetaData());
</pre><br />
This declares that the database schema is to be mirrored into the com.myproject.domain package in the src/main/java folder.<br />
<br />
The generated types have the table name transformed to mixed case as the class name and a similar mixed case transformation applied to the columns which are available as property paths in the query type.<br />
<br />
<h4>In Maven</h4><br />
This functionality is also available as a Maven plugin. The presented example can be declared like this in the POM :<br />
<br />
<pre name="code" class="brush:xml"><plugin>
<groupId>com.mysema.querydsl</groupId>
<artifactId>querydsl-maven-plugin</artifactId>
<version>${querydsl.version}</version>
<executions>
<execution>
<goals>
<goal>export</goal>
</goals>
</execution>
</executions>
<configuration>
<jdbcDriver>org.apache.derby.jdbc.EmbeddedDriver</jdbcDriver>
<jdbcUrl>jdbc:derby:target/demoDB;create=true</jdbcUrl>
<packageName>com.myproject.domain</packageName>
<targetFolder>${project.basedir}/target/generated-sources/java</targetFolder>
</configuration>
<dependencies>
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derby</artifactId>
<version>${derby.version}</version>
</dependency>
</dependencies>
</plugin>
</pre><br />
Use the goal test-export to add the targetFolder as a test compile source root instead of a compile source root.<br />
<br />
<h4>In ANT</h4><br />
The ANT task <b>com.mysema.query.sql.ant.AntMetaDataExporter</b> of the querydsl-sql module provides the same functionality as an ANT task. The configuration parameters of the task are jdbcDriverClass, dbUrl, dbUserName, dbPassword, namePrefix, targetPackage, targetSourceFolder, schemaPattern, tableNamePattern, exportBeans and innerClassesForKeys. <br />
<br />
<h3>Using query types</h3><br />
To create queries with Querydsl you need to instantiate variables and Query implementations. We will start with the variables.<br />
<br />
For each table in your mirrored schema a query type is created. For example for the table <b>customer</b> Querydsl will generate a query type with the simple name <b>QCustomer</b> into the folder and package declared in the code generation setup. <b>QCustomer</b> can be used as a statically typed variable in Querydsl queries as a representative for the customer table.<br />
<br />
<b>QCustomer</b> has a default instance variable which can be accessed as a static field :<br />
<br />
<pre name="code" class="brush:java">QCustomer customer = QCustomer.customer;
</pre><br />
Alternatively you can define your own <b>QCustomer</b> instances like this :<br />
<br />
<pre name="code" class="brush:java">QCustomer customer = new QCustomer("c");
</pre><br />
<br />
<h3>Querying</h3><br />
Querying with Querydsl SQL is as simple as this :<br />
<br />
<pre name="code" class="brush:java">QCustomer customer = new QCustomer("c"); // alias for the CUSTOMER table
SQLTemplates dialect = new HSQLDBTemplates(); // SQL-dialect
SQLQuery query = new SQLQueryImpl(connection, dialect);
List<String> lastNames = query.from(customer)
.where(customer.firstName.eq("Bob"))
.list(customer.lastName);
</pre><br />
which is transformed into the following sql query, assuming that the related table name is customer and the columns first_name and last_name :<br />
<br />
<pre name="code" class="brush:sql">SELECT c.last_name
FROM customer c
WHERE c.first_name = 'Bob'
</pre><br />
Internally Querydsl SQL uses prepared statements.<br />
<br />
<h3>General usage</h3><br />
Use the the cascading methods of the <b>com.mysema.query.sql.SQLQuery</b> interface like this<br />
<br />
<b>from</b> : Define the query sources here.<br />
<br />
<b>innerJoin, join, leftJoin, fullJoin, on</b> : Define join elements using these constructs. For the join methods the first argument is the join source and the second the target (alias).<br />
<br />
<b>where</b> : Define the query filters, either in varargs form separated via commas or cascaded via the and-operator.<br />
<br />
<b>groupBy</b> : Define the group by arguments in varargs form.<br />
<br />
<b>having</b> : Define the having filter of the "group by" grouping as an varags array of <b>Predicate</b> expressions.<br />
<br />
<b>orderBy</b> : Define the ordering of the result as an varargs array of order expressions. Use <b>asc()</b> and <b>desc()</b> on numeric, string and other comparable expression to access the OrderSpecifier instances.<br />
<br />
<b>limit, offset, restrict</b> : Define the paging of the result. Limit for max results, offset for skipping rows and restrict for defining both in one call.<br />
<br />
<h3>Ordering</h3><br />
The syntax for declaring ordering is<br />
<br />
<pre name="code" class="brush:java">query.from(customer)
.orderBy(customer.lastName.asc(), customer.firstName.asc())
.list(customer.firstName, customer.lastName);
</pre><br />
which is equivalent to the following native SQL<br />
<br />
<pre name="code" class="brush:sql">SELECT c.first_name, c.last_name
FROM customer c
ORDER BY c.last_name ASC, c.first_name ASC
</pre><br />
<h3>Grouping</h3><br />
Grouping can be done in the following form<br />
<br />
<pre name="code" class="brush:java">query.from(customer)
.groupBy(customer.lastName)
.list(customer.lastName);
</pre><br />
which is equivalent to the following native SQL<br />
<br />
<pre name="code" class="brush:sql">SELECT c.last_name
FROM customer c
GROUP BY c.last_name
</pre><br />
<h3>Subqueries</h3><br />
To create a subquery you create a <b>com.mysema.query.sql.SQLSubQuery</b> instance, define the query parameters via from, where etc and use unique or list to create a subquery, which is just a type-safe Querydsl expression for the query. unique is used for a unique (single) result and list for a list result.<br />
<br />
<pre name="code" class="brush:java">query.from(customer).where(
customer.status.eq(new SQLSubQuery().from(customer2).unique(customer2.status.max()))
.list(customer.all())
</pre><br />
Another example<br />
<br />
<pre name="code" class="brush:java">query.from(customer).where(
customer.status.in(new SQLSubQuery().from(status).where(status.level.lt(3)).list(status.id))
.list(customer.all())
</pre><br />
<h3>Query extension support</h3><br />
Custom query extensions to support engine specific syntax can be created by subclassing <b>com.mysema.query.sql.AbstractSQLQuery</b> and adding flagging methods like in the given MySQLQuery example :<br />
<br />
<pre name="code" class="brush:java">public class MySQLQuery extends AbstractSQLQuery<MySQLQuery>{
public MySQLQuery(Connection conn) {
this(conn, new MySQLTemplates(), new DefaultQueryMetadata());
}
public MySQLQuery(Connection conn, SQLTemplates templates) {
this(conn, templates, new DefaultQueryMetadata());
}
protected MySQLQuery(Connection conn, SQLTemplates templates, QueryMetadata metadata) {
super(conn, new Configuration(templates), metadata);
}
public MySQLQuery bigResult(){
return addFlag(Position.AFTER_SELECT, "SQL_BIG_RESULT ");
}
public MySQLQuery bufferResult(){
return addFlag(Position.AFTER_SELECT, "SQL_BUFFER_RESULT ");
}
// ...
}
</pre><br />
<br />
The flags are custom SQL snippets that can be inserted at specific points in the serialization. The supported positions are the enums of the <b>com.mysema.query.QueryFlag.Position</b> enum class.<br />
<br />
<h3>Using DDL commands</h3><br />
<b>CREATE TABLE</b> commands can be used in fluent form via the <b>com.mysema.query.sql.ddl.CreateTableClause</b> class. Here are some examples :<br />
<br />
<pre name="code" class="brush:java">new CreateTableClause(conn, templates, "language")
.column("id", Integer.class).notNull()
.column("text", String.class).size(256).notNull()
.primaryKey("PK_LANGUAGE","id")
.execute();
new CreateTableClause(conn, templates, "symbol")
.column("id", Long.class).notNull()
.column("lexical", String.class).size(1024).notNull()
.column("datatype", Long.class)
.column("lang", Integer.class)
.column("intval",Long.class)
.column("floatval",Double.class)
.column("datetimeval",Timestamp.class)
.primaryKey("PK_SYMBOL","id")
.foreignKey("FK_LANG","lang").references("language","id")
.execute();
new CreateTableClause(conn, templates, "statement")
.column("model", Long.class)
.column("subject", Long.class).notNull()
.column("predicate", Long.class).notNull()
.column("object", Long.class).notNull()
.foreignKey("FK_MODEL","model").references("symbol","id")
.foreignKey("FK_SUBJECT","subject").references("symbol","id")
.foreignKey("FK_PREDICATE","predicate").references("symbol","id")
.foreignKey("FK_OBJECT","object").references("symbol","id")
.execute();
</pre><br />
The constructor of <b>CreateTableClause</b> takes the connection, the templates and the table name. The rest is declared via <b>column</b>, <b>primaryKey</b> and <b>foreignKey</b> invocations.<br />
<br />
Here are the corresponding <b>CREATE TABLE</b> clauses as they are executed.<br />
<br />
<pre name="code" class="brush:sql">CREATE TABLE language (
id INTEGER NOT NULL,
text VARCHAR(256) NOT NULL,
CONSTRAINT PK_LANGUAGE PRIMARY KEY(id)
)
CREATE TABLE symbol (
id BIGINT NOT NULL,
lexical VARCHAR(1024) NOT NULL,
datatype BIGINT,
lang INTEGER,
intval BIGINT,
floatval DOUBLE,
datetimeval TIMESTAMP,
CONSTRAINT PK_SYMBOL PRIMARY KEY(id),
CONSTRAINT FK_LANG FOREIGN KEY(lang) REFERENCES language(id)
)
CREATE TABLE statement (
model BIGINT,
subject BIGINT NOT NULL,
predicate BIGINT NOT NULL,
object BIGINT NOT NULL,
CONSTRAINT FK_MODEL FOREIGN KEY(model) REFERENCES symbol(id),
CONSTRAINT FK_SUBJECT FOREIGN KEY(subject) REFERENCES symbol(id),
CONSTRAINT FK_PREDICATE FOREIGN KEY(predicate) REFERENCES symbol(id),
CONSTRAINT FK_OBJECT FOREIGN KEY(object) REFERENCES symbol(id)
)
</pre><br />
<h3>Using DML commands</h3><br />
All the <b>com.mysema.query.dml.DMLClause</b> implementations in the Querydsl SQL module take three parameters, the <b>java.sql.Connection</b> instance, the <b>com.mysema.query.sql.SQLTemplates</b> instance used in the queries and the main entity the <b>DMLClause</b> is bound to.<br />
<br />
Insert examples :<br />
<br />
<pre name="code" class="brush:java">// with columns
new SQLInsertClause(conn, dialect, survey)
.columns(survey.id, survey.name)
.values(3, "Hello").execute();
// without columns
new SQLInsertClause(conn, dialect, survey)
.values(4, "Hello").execute();
// with subquery
new SQLInsertClause(conn, dialect, survey)
.columns(survey.id, survey.name)
.select(new SQLSubQuery().from(survey2).list(survey2.id.add(1), survey2.name))
.execute();
// with subquery, without columns
new SQLInsertClause(conn, dialect, survey)
.select(new SQLSubQuery().from(survey2).list(survey2.id.add(10), survey2.name))
.execute();
</pre><br />
Update examples : <br />
<br />
<pre name="code" class="brush:java">// update with where
new SQLUpdateClause(conn, dialect, survey)
.where(survey.name.eq("XXX"))
.set(survey.name, "S")
.execute();
// update without where
new SQLUpdateClause(conn, dialect, survey)
.set(survey.name, "S")
.execute()
</pre><br />
Delete examples :<br />
<br />
<pre name="code" class="brush:java">// delete with where
new SQLDelecteClause(conn, dialect, survey)
.where(survey.name.eq("XXX"))
.execute();
// delete without where
new SQLDelecteClause(conn, dialect, survey)
.execute()
</pre><br />
<h3>Bean class generation</h3><br />
To create JavaBean DTO types for the tables of your schema use the <b>com.mysema.query.sql.MetaDataExporter</b> like this :<br />
<br />
<pre name="code" class="brush:java">java.sql.Connection conn = ...;
MetaDataExporter exporter = new MetaDataExporter();
exporter.setPackageName("com.myproject.mydomain");
exporter.setTargetFolder(new File("src/main/java"));
exporter.setBeanSerializer(new BeanSerializer());
exporter.export(conn.getMetaData());
</pre><br />
Now you can use the bean types as arguments to the populate method in DML clauses and you can project directly to bean types in queries. Here is a simple example :<br />
<br />
<pre name="code" class="brush:java">@Test
QEmployee e = new QEmployee("e");
// Insert
Employee employee = new Employee();
employee.setFirstname("John");
Integer id = insert(e).populate(employee).executeWithKey(e.id);
employee.setId(id);
// Update
employee.setLastname("Smith");
assertEquals(1l, update(e).populate(employee).where(e.id.eq(employee.getId())).execute());
// Query
Employee smith = query().from(e).where(e.lastname.eq("Smith")).uniqueResult(e);
assertEquals("John", smith.getFirstname());
// Delete
assertEquals(1l, delete(e).where(e.id.eq(employee.getId())).execute());
</pre><br />
<h3>Where to go next</h3><br />
See the related section of the latest <a href="http://source.mysema.com/static/querydsl/2.1.0-beta2/reference/html/ch02s04.html">Reference docs</a> for more information.Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com12tag:blogger.com,1999:blog-8954277266683132656.post-88569066629074397562010-11-10T12:06:00.015+02:002012-08-21T11:38:15.613+03:00MongoDB with Querydsl<a href="http://www.querydsl.com">Querydsl</a> provides a typesafe querying layer on top of JPA, JDO, JDBC and other backends. This blog post presents a simple tutorial on how to get started with querying on <a href="http://www.mongodb.org/">MongoDB</a> using Querydsl.<br />
<br />
<h3>Maven integration</h3><br />
To get started with Querydsl for MongoDB using a Maven 2 based build environment, follow the following steps.<br />
<br />
Add the following dependencies to your Maven project and make sure that the <a href="http://source.mysema.com/maven2/releases">Maven 2 repo of Mysema Source</a> is accessible from your POM :<br />
<br />
<pre name="code" class="brush:xml"><dependency>
<groupId>com.mysema.querydsl</groupId>
<artifactId>querydsl-mongodb</artifactId>
<version>${querydsl.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.6.1</version>
</dependency>
</pre><br />
Querydsl uses the Annotation Processing Tool of Java 6 for code generation which needs to be configured as well :<br />
<br />
<pre name="code" class="brush:xml"><project>
<build>
<plugins>
...
<plugin>
<groupId>com.mysema.maven</groupId>
<artifactId>maven-apt-plugin</artifactId>
<version>1.0</version>
<executions>
<execution>
<goals>
<goal>process</goal>
</goals>
<configuration>
<outputDirectory>${generatedSources}<outputDirectory>
<processor>com.mysema.query.mongodb.morphia.MorphiaAnnotationProcessor</processor>
</configuration>
</execution>
</executions>
</plugin>
...
</plugins>
</build>
</project>
</pre><br />
The <i>MorphiaAnnotationProcessor</i> finds domain types annotated with the <i>com.google.code.morphia.annotations.Entity</i> and <i>com.google.code.morphia.annotations.Embedded</i> annotations and generates query types for them.<br />
<br />
Run clean install and you will get your Query types generated into <i>target/generated-sources/java</i>.<br />
<br />
If you use Eclipse, run mvn eclipse:eclipse to update your Eclipse project to include <i>target/generated-sources/java</i> as a source folder.<br />
<br />
Now you are able to construct Querydsl query instances and instances of the query domain model.<br />
<br />
<br />
<h3>Ant integration</h3><br />
If you have an Ant based build environment you can use one of the packaged releases from the <a href="http://source.mysema.com/display/querydsl/Downloads">Downloads page</a>.<br />
<br />
Place the jar files from the full-deps bundle on your classpath and use the following tasks for Querydsl code generation :<br />
<br />
<pre name="code" class="brush:xml"><!-- APT based code generation -->
<javac srcdir="${src}" classpathref="cp">
<compilerarg value="-proc:only"/>
<compilerarg value="-processor"/>
<compilerarg value="com.mysema.query.mongodb.morphia.MorphiaAnnotationProcessor"/>
<compilerarg value="-s"/>
<compilerarg value="${generated}"/>
</javac>
<!-- compilation -->
<javac classpathref="cp" destdir="${build}">
<src path="${src}"/>
<src path="${generated}"/>
</javac>
</pre><br />
Replace <i>src</i> with your main source folder, <i>generated</i> with your folder for generated sources and <i>build</i> with your target folder.<br />
<br />
<h3>Using query types</h3><br />
To create queries with Querydsl you need to instantiate variables and Query implementations. We will start with the variables.<br />
<br />
Let's assume that your project has the following domain type :<br />
<br />
<pre name="code" class="brush:java">@Entity
public class Customer {
private @Id ObjectId id;
private String firstName;
private String lastName;
public String getFirstName(){
return firstName;
}
public String getLastName(){
return lastName;
}
public void setFirstName(String fn){
firstName = fn;
}
public void setLastName(String ln)[
lastName = ln;
}
public ObjectId getId() {
return id;
}
public void setId(ObjectId id) {
this.id = id;
}
}
</pre><br />
Querydsl will generate a query type with the simple name QCustomer into the same package as Customer. QCustomer can be used as a statically typed variable in Querydsl queries as a representative for the Customer type.<br />
<br />
QCustomer has a default instance variable which can be accessed as a static field :<br />
<br />
<pre name="code" class="brush:java">QCustomer customer = QCustomer.customer;
</pre><br />
Alternatively you can define your own Customer variables like this :<br />
<br />
<pre name="code" class="brush:java">QCustomer customer = new QCustomer("myCustomer");
</pre><br />
<h3>Querying</h3><br />
For the MongoDB-module MorphiaQuery is the main Query implementation. It is instantiated like this :<br />
<br />
<pre name="code" class="brush:java">// test database
String dbname = "testdb";
// Morphia configuration
Morphia morphia = new Morphia().map(Customer.class);
// datastore
Datastore ds = morphia.createDatastore(dbname);
QCustomer customer = QCustomer.customer;
MorphiaQuery<Customer> query = new MorphiaQuery<Customer>(morphia, ds, customer);
</pre><br />
To retrieve the customer with the first name Bob you would construct a query like this :<br />
<br />
<pre name="code" class="brush:java">QCustomer customer = QCustomer.customer;
MorphiaQuery<Customer> query = new MorphiaQuery<Customer>(morphia, ds, customer);
Customer bob = query.where(customer.firstName.eq("Bob"))
.uniqueResult();
</pre><br />
The where part defines the filter and uniqueResult tells Querydsl to return a single element. Easy, right?<br />
<br />
And to use multiple filters use it like this<br />
<br />
<pre name="code" class="brush:java">query
.where(customer.firstName.eq("Bob"), customer.lastName.eq("Wilson"));
</pre><br />
Or like this<br />
<br />
<pre name="code" class="brush:java">query
.where(
customer.firstName.eq("Bob").and(customer.lastName.eq("Wilson")));
</pre><br />
<br />
<h3>General usage</h3><br />
Use the the cascading methods of the MorphiaQuery class like this<br />
<br />
<ul><li>where : Define the query filters, either in varargs form separated via commas or cascaded via the and-operator.</li>
<li>orderBy : Define the ordering of the result as an varargs array of order expressions. Use asc() and desc() on numeric, string and other comparable expression to access the OrderSpecifier instances.</li>
<li>limit, offset, restrict : Define the paging of the result. Limit for max results, offset for skipping rows and restrict for defining both in one call.</li>
</ul><br />
<h3>Ordering</h3><br />
The syntax for declaring ordering is<br />
<br />
<pre name="code" class="brush:java">query
.orderBy(customer.lastName.asc(), customer.firstName.desc())
.list();
</pre><br />
<br />
<h3>Where to go next</h3><br />
For an up-to-date guide on how to use Querydsl with MongoDB, see the latest <a href="http://source.mysema.com/display/querydsl/Documentation">Reference Documentation</a>.<br />
<br />
And if you have any issues, questions or ideas for enhancement post a thread in the <a href="http://source.mysema.com/forum/mvnforum/index">Mysema Source Forum</a>.<br />
<br />
Happy querying!Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com4tag:blogger.com,1999:blog-8954277266683132656.post-72957412913080000142010-10-04T13:18:00.012+03:002012-03-09T18:00:14.478+02:00SQL queries in ScalaIn a previous blog post I showed you how <a href="http://blog.mysema.com/2010/09/querying-with-scala.html">Querydsl and Scala</a> works with JPA, JDO and Mongodb. This blog post shows you how to use Scala queries on top of Querydsl SQL.<br />
<br />
If you are using Maven you need the following two dependencies : <br />
<br />
<pre name="code" class="brush:xml"><dependency>
<groupId>com.mysema.querydsl</groupId>
<artifactId>querydsl-scala</artifactId>
<version>${querydsl.version}</version>
</dependency>
<dependency>
<groupId>com.mysema.querydsl</groupId>
<artifactId>querydsl-sql</artifactId>
<version>${querydsl.version}</version>
</dependency>
</pre><br />
Like with Querydsl SQL for Java you need to generate Query types to be able to construct your queries. The following two unit test methods show how this is done :<br />
<br />
<pre name="code" class="brush:java">val templates = new HSQLDBTemplates()
var connection: Connection = _
@Before
def setUp() {
// set up the JDBC connection for JDBC metadata access
Class.forName("org.hsqldb.jdbcDriver")
val url = "jdbc:hsqldb:mem:testdb"
connection = DriverManager.getConnection(url, "sa", "")
}
@Test
def Generation_without_Beantypes() {
val namingStrategy = new DefaultNamingStrategy()
val serializer = new ScalaMetaDataSerializer("Q", namingStrategy)
val exporter = new MetaDataExporter("Q", "test", new File("target/gen1"), namingStrategy, serializer)
exporter.setCreateScalaSources(true)
exporter.export(connection.getMetaData())
}
@Test
def Generation_with_Beantypes() {
val namingStrategy = new DefaultNamingStrategy()
val serializer = new ScalaMetaDataSerializer("Q", namingStrategy)
val beanSerializer = new ScalaBeanSerializer()
val exporter = new MetaDataExporter("Q", "com.mysema.example", new File("target/gen2"), namingStrategy, serializer, beanSerializer)
exporter.setCreateScalaSources(true)
exporter.export(connection.getMetaData())
}
</pre><br />
The MetaDataExporter can be used with and without Bean type generation. If you just want to write SQL queries and you don't need a JavaBean based model for projection, insertion and update, you can use it without bean projection. If you want to mirror your SQL schema into a minimal domain model and use it ORM-like then generate the bean types.<br />
<br />
Make sure that the target folder (target/gen1 and target/gen2 in the two examples) is in your source path. And replace com.mysema.example with a target package of your choice. <br />
<br />
Now you can use the generated query types for querying :<br />
<br />
<pre name="code" class="brush:java">@Test
def Querying() {
val survey = new QSurvey("survey")
val employee = new QEmployee("employee")
assertEquals("abc", query from survey where (survey.id eq 1)
.uniqueResult survey.name)
assertEquals("def", query from survey where (survey.id eq 2)
.uniqueResult survey.name)
assertEquals("Bob", query from employee
.where (employee.lastname eq "Smith")
.uniqueResult employee.firstname)
assertEquals("John", query from employee
.where (employee.lastname eq "Doe")
.uniqueResult employee.firstname)
}
def query = new SQLQueryImpl(connection, templates)
</pre><br />
Querydsl SQL for Scala doesn't use the $-sign based escape syntax for expression construction nor alias variables. Instead code generation is used to output the Query model as a set of Scala classes that reflect the tables, columns and keys of your relational schema.<br />
<br />
You can instantiate the generated Q-types instead. Alternatively you can also use a more SQL-like syntax:<br />
<br />
<pre name="code" class="brush:java">val survey = QSurvey as "survey"
</pre><br />
For more details on how to use Querydsl SQL see the related section in the latest revision of the <a href="http://source.mysema.com/static/querydsl/latest/reference/html/">Reference Documentation</a>.<br />
<br />
Recent released have changed the Querydsl Scala API. Visit the Reference Documentation for updated documentation.Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com0tag:blogger.com,1999:blog-8954277266683132656.post-29473902449679011462010-10-01T15:02:00.005+03:002010-10-01T15:24:49.527+03:00Using Querydsl In EclipseQuerydsl usage in Eclipse is now easier than ever. Since Querydsl 2.0 we have improved APT support in Querydsl to support incremental code generation in IDEs. We have also improved the release bundles to streamline the APT configuration in Java IDEs like Eclipse. <br />
<br />
The APT usage improvements affect the JPA, JDO and MongoDB modules. Instead of a set of screenshots we have prepared a screencast to showcase the new APT features in Querydsl in an Eclipse context.<br />
<br />
To enable APT usage in Eclipse you need to <br />
<ul><li>set the <b>Enable annotation processing</b> checkbox via <b>Properties > Java Compiler > Annotation Processing</b>,</li>
<li>add the APT option <b>defaultOverwrite = true</b> if you are using Eclipse Helios and</li>
<li>set the Factory Path to one of the apt-one-jar files included in your release bundle.</li>
</ul><br />
The screencast uses the Querydsl JPA module in combination with Hibernate and shows the creation of a Java project, the creation of two simple domain types and the creation of a minimal demo service which presents the simplicity of Querydsl query usage.<br />
<br />
<object width="480" height="385"><param name="movie" value="http://www.youtube.com/v/Ef5Kl_5RP-0?fs=1&hl=en_US"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/Ef5Kl_5RP-0?fs=1&hl=en_US" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="480" height="385"></embed></object>Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com5tag:blogger.com,1999:blog-8954277266683132656.post-34990693536955011862010-09-28T16:45:00.017+03:002013-05-08T16:25:22.089+03:00Querydsl 2.0 has been released!Querydsl 2.0 has been released! After two alpha and two beta releases we are proud to announce the final 2.0 release. The biggest single change in Querydsl 2.0 compared to previous 1.* releases is the improved Expression API. We got rid of the coupling between the query construction contract and the DSL features of the Expression API. <br />
<br />
The Expression API is now divided into two parts, an interface based part which describes the archetypes of Expressions which are used in Query construction, and a class based part, which provides the standard Expression DSL.<br />
<br />
Here is an UML overview of the new Expression model : <a href="http://source.mysema.com/forum/mvnforum/viewthread_thread,158">Expression model changes</a><br />
<br />
In addition to the improvements of the Expression API there is improved <a href="http://www.eclipse.org/">Eclipse IDE</a> support, initial <a href="http://www.scala-lang.org/">Scala</a> and <a href="http://www.mongodb.org/">Mongodb</a> support.Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com0tag:blogger.com,1999:blog-8954277266683132656.post-19495699285451278872010-09-28T16:45:00.016+03:002013-05-08T16:25:40.386+03:00Querying with ScalaScala support for Querydsl has been on the roadmap for some time, and with the Querydsl 2.0 release we got initial support for using Querydsl in Scala applications released.<br />
<br />
The goal was to use the Query interfaces as such, and to provide an alternative expression DSL better suited for Scala usage. With most Querydsl modules APT is used for code generation. As there is no equivalent for Scala we didn't try to provide code generation features for Scala domain models. <br />
<br />
Instead we combined the Alias-feature of the Querydsl Core module with the Scala expression DSL. The result is in our opinion quite compact and readable.<br />
<br />
For the moment Querydsl with Scala can be used with JPA, JDO and Mongodb. To get started, you will need the additional querydsl-scala dependency in your project in addition to the dependencies for your Querydsl backend support. Querydsl for Scala has been tested against Scala 2.8.0.<br />
<br />
Using Maven you would need to declare the following additional dependency :<br />
<br />
<pre name="code" class="brush:xml"><dependency>
<groupId>com.mysema.querydsl</groupId>
<artifactId>querydsl-scala</artifactId>
<version>${querydsl.version}</version>
</dependency>
</pre><br />
Here is a minimal example with JPA/Hibernate :<br />
<br />
Domain model : <br />
<br />
<pre name="code" class="brush:java">@Entity
class User {
@BeanProperty
@Id
var id: Integer = _;
@BeanProperty
var userName: String = _;
@BeanProperty
@ManyToOne
var department: Department = _;
}
@Entity
class Department {
@BeanProperty
@Id
var id: Integer = _;
@BeanProperty
var name: String = _;
}
</pre><br />
And here are some query examples <br />
<br />
<pre name="code" class="brush:java">import com.mysema.query.scala.Conversions._
import com.mysema.query.jpa.impl.JPAQuery
import com.mysema.query.types.path._
import org.junit.Test
class JPAQueryTest {
val person = alias(classOf[Person])
@Test
def Various() {
// list
query from person where (person.firstName $like "Rob%")
.list person
// unique result
query from person where (person.firstName $like "Rob%")
.unique person
// long where
query from person
.where (person.firstName $like "Rob%", person.lastName $like "An%")
.list person
// order
query from person orderBy (person.firstName asc) list person
// not null
query from person
.where (person.firstName $isEmpty, person.lastName $isNotNull)
.list person
}
def query() = new JPAQuery(entityManager)
}
</pre><br />
The main import for Querydsl Scala integration is the following<br />
<br />
<pre name="code" class="brush:java">import com.mysema.query.scala.Conversions._
</pre><br />
The factory method for query creation is <br />
<br />
<pre name="code" class="brush:java">def query() = new JPAQuery(entityManager)
</pre><br />
In addition to queries you need variables which can be created like this<br />
<br />
<pre name="code" class="java">var person = alias(classOf[Person])
</pre><br />
The person variable is a proxy instance of the the Person class which can be used in queries. Now you can construct your queries, populate them via from-where-...-orderBy calls and get the projection out via list/uniqueResult/listResults calls.<br />
<br />
Querydsl expressions are constructed via method calls starting with the "$" sign.<br />
<br />
With the Querydsl Java API a simple like expression would be constructed like this :<br />
<br />
<pre name="code" class="brush:java">person.firstName.like("Rob%")
</pre><br />
Using the Scala API it is <br />
<br />
<pre name="code" class="brush:java">person.firstName $like "Rob%"
</pre><br />
For more examples see src/test/scala in <a href="https://source.mysema.com/svn/mysema/projects/querydsl/trunk/querydsl-scala/">SVN</a>.Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com0tag:blogger.com,1999:blog-8954277266683132656.post-16005192615012984042010-09-10T10:09:00.012+03:002010-09-10T12:51:36.326+03:00Secure Registration MethodMany services require users to register using real email addresses. Usually this is done so that the system sends a confirmation message to the email address after the user has registered. This requires a two phase flow in which the account details are first persisted into an inactive state and later activated. An alternative is to make the account temporarily active to let the user in right away, but close it after a while if there is no email confirmation.<br /><br />There are several drawbacks to this two phase registration method:<br /><ol><li>Increased complexity in handling account activation</li><br /><li>Unconfirmed users are persisted and need to be cleaned up from the database from time to time</li><br /><li>If the user is let in temporarily right away, there have to be additional measures to guard against bots and to authorize actions that are reserved for actually confirmed accounts. </li><br /><li>It is very inconvenient for the user if she first gives all her details and then somehow fails to confirm the account, e.g. due to a typo in the email address.</li></ol><br />A much simpler, yet secure way of registration starts by confirming the email address. First the user gives her email address and the system sends her an email containing a unique registration link. With a properly constructed link, nothing needs to be persisted on the server side at this point. Only when the user follows the link, is she requested for other account details. The account is then created directly into an active state and the user is let in with full access rights.<br /><br />The trick is in the link itself. It should be constructed to be a) hard to guess and tamper with and b) expiring by itself after a fixed amount of time. This can be achieved easily by adding a timestamp and the email address itself as parameters to the link and using a <a href="http://en.wikipedia.org/wiki/Message_authentication_code">Message Authentication Code</a> (MAC) to ensure that these parameters are not tampered with:<br /><pre>https://yourdomain.com/register<b><br />?</b>timestamp=123456789<b>&</b>email=john.doe@example.com<b>&</b>mac=MAC</pre> in which the MAC is constructed using a hash function like SHA:<br /><pre>sha(123456789;john.doe@example.com;SECRET)</pre> Here the SECRET is a long random phrase that is only known by the server. Even if one knows the building blocks of the link, one cannot fake it unless one knows the secret phrase.<br /><br />Before even showing the user a registration form, the MAC is to be checked by comparing the supplied MAC with one calculated from the other message parameters.<br /><br />Once we know that we can trust the link parameters we know that the given email address exists and is accessible by the user. Expiring the link is simply a matter of checking whether the timestamp is not too far in the past (e.g. three hours).Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-8954277266683132656.post-49776297891343449932010-09-06T12:55:00.003+03:002010-09-10T12:59:22.703+03:00Software business would be so easy without customers and usersThis has sometimes come to my mind when being in one of those moments of pressure to deliver the promised features on time. Or when hearing that the delivery is not what was ordered or doesn’t have all the features imagined or is perfect apart from being late.<br /><br />Well, no-one says the software business is easy, and it definitely isn’t so. But why is creating working software in the real world so difficult? I think the main reason is because the software business is about creating and selling something which is much more intangible than what we as human beings are used to deal with. And we are pretty much just learning how this new field of industry really works.<br /><br />The intangible nature of the software can be seen from how<br /><ul><li>things can change fast</li><li>big changes are possible</li><li>it’s not easy to describe what is actually needed</li><li>it’s not easy to say if something is good enough</li><li>there is always multiple workable ways to solve problems<br /></li></ul><br />The rules and the laws on how to build a good house and good software are different, we have a long history on successfully constructing the buildings, we know how to do it well, but we are just learning on what kind of new skills and ways of thinking are needed to create successful software projects.<br /><br /><h3>Embracing the change </h3><br />Using agile methods to create software comes from the learned lesson that it’s not useful to fight against the intangible nature of software and the process of developing it. We have to have tools that are able embrace the change rather than fight against it. And the whole iterative development process with practices like continuous integration, delivering often, test driven development and so on are aimed to make that possible.<br /><br />So now we have a toolbox full of good ways to react fast when things go into unexpected directions as they always do and everything is fine, right?<br /><br />Well, no. What hasn’t changed that much at all is the relationship between the customers and developers. While the methods of creating the software have changed to be more agile, the contracts have not changed that much. And in the end the contract is the bottom line of what is possible in the relationship between these two parties.<br /><br />The tighter the contract, the less it allows room to react, and the more there is risk that the whole project is doomed in the end due to trying to pretend that what everyone is doing is something solid, pre-designable and predictable far in advance.<br /><br /><h3>The trust allows more space to react</h3><br />Constructing software for someone is about fusing their ideas and wishes into working software. And if it would only be that, it would be quite manageable, but unfortunately there is no escape from the hard facts of time and money, and usually in the form of not having enough of both.<br /><br />Agility means the ability to change the direction fast, the steer of the boat of software creation. And changing direction means there has to be some openness that the current direction might not be the best direction. So being agile means inherently being able to withstand a little bit of unsureness, the lack of knowing. The more we can feel confident in moving to uncharted territory with the knowledge that if things go bad, we can react correctly, the faster we can travel safely. But that’s confidence on the tools and the people you are traveling with, it’s not the (contractually) enforced confidence in the plan that was created in the past and which might not be working very well anymore in the changed situation.<br /><br />So, in order for the software process to be truly agile, there has to be trust between the parties. Trust that both are actually traveling together, not a totally known path, but mutually agreed direction. When this trust is not present, what remains are the rigid steel frames of contracts. And usually this state also contracts the space of creativity too.<br /><br /><h3>The big questions</h3><br />Now that we have some ideas on what is a good way to create software and what kind of relationship between a paying customer and supplier would be good base for the successful project, there are two big questions:<br /><ul><li>How do you sell an agile software project?</li><li>How do you create a good contract for an agile project?<br /></li></ul><br />I’m positive that there is plenty of room for improvements in that arena and it’s going to be interesting to see how the agile movement is going to be seen in the arena of selling and making contracts. I will try to cover some ideas on them in my next blog post.Lassi Immonenhttp://www.blogger.com/profile/01836670514186979536noreply@blogger.com0tag:blogger.com,1999:blog-8954277266683132656.post-50629723346043527512010-07-31T10:48:00.004+03:002013-05-08T16:26:02.969+03:00Query Extensions in ActionQuerydsl is all about providing the best type-safe querying tool for Java. To be the best fit for most of the situations, Querydsl provides lots of customization possibilities. <br />
<br />
Query types can be extended to fit custom querying needs, custom method templates can be added to generated types and static delegate methods can be used as well.<br />
<br />
Until recently static delegate methods could only be used for entity types. Since version 1.8.3 they can be used for built-in types as well.<br />
<br />
This feature was requested by <a href="http://luisfpg.blogspot.com/">Luis Fernando Planella Gonzalez</a> from the <a href="http://project.cyclos.org/">Cyclos</a> project and they immediately took it into use.<br />
<br />
Here is the declaration of their delegate methods :<br />
<br />
<pre name="code" class="brush:java">/**
* Contains extensions for common query classes
* @author luis
*/
public class QueryExtensions {
/**
* Adds a period filter
*/
@QueryDelegate(Date.class)
public static EBoolean period(PDate<Date> expr, IDatePeriod period) {
BooleanBuilder bb = new BooleanBuilder();
Date begin = period == null ? null : period.getBegin();
if (begin != null) {
bb.and(expr.goe(begin));
}
Date end = period == null ? null : period.getEnd();
if (end != null) {
bb.and(expr.loe(end));
}
return bb;
}
/**
* Adds a date period filter on a timestamp expression
*/
@QueryDelegate(Timestamp.class)
public static EBoolean period(PDateTime<Timestamp> expr, IDatePeriod period) {
BooleanBuilder bb = new BooleanBuilder();
Date begin = period == null ? null : period.getBegin();
if (begin != null) {
bb.and(expr.goe(DateHelper.toTime(begin)));
}
Date end = period.getEnd();
if (end != null) {
bb.and(expr.lt(DateHelper.toTime(DateHelper.truncateNextDay(end))));
}
return bb;
}
/**
* Adds a timestamp period filter on a timestamp expression
*/
@QueryDelegate(Timestamp.class)
public static EBoolean period(PDateTime<Timestamp> expr, ITimestampPeriod period) {
BooleanBuilder bb = new BooleanBuilder();
Timestamp begin = period == null ? null : period.getBegin();
if (begin != null) {
bb.and(expr.goe(begin));
}
Timestamp end = period.getEnd();
if (end != null) {
bb.and(expr.loe(end));
}
return bb;
}
}
</pre><br />
All of the shown delegate methods are factory methods for "date in period" style expressions. The types for date and period vary in the methods.<br />
<br />
The semantics of QueryDelegate annotations are simple. The @QueryDelegate.value determines the real type for the expression type to be extended and the first argument of the delegate method is used for the expression type instance itself. The rest of the arguments are used for the method in the extended query type.<br />
<br />
In this example extensions for the types <i>java.sql.Date</i> and <i>java.sql.Timestamp</i> are declared. The respective builtin types in Querydsl are PDate and PDateTime. Two classes are created to extend the signature for PDate and PDateTime for Date and Timestamp usage :<br />
<br />
<pre name="code" class="brush:java">public class QDate extends PDate<java.sql.Date> {
private static final long serialVersionUID = 494982445;
public QDate(PEntity<? extends java.sql.Date> entity) {
super(entity.getType(),entity.getMetadata());
}
public QDate(PathMetadata<?> metadata) {
super(java.sql.Date.class, metadata);
}
public EBoolean period(IDatePeriod period) {
return QueryExtensions.period(this, period);
}
}
public class QTimestamp extends PDateTime<java.sql.Timestamp> {
private static final long serialVersionUID = 789961463;
public QTimestamp(PEntity<? extends java.sql.Timestamp> entity) {
super(entity.getType(),entity.getMetadata());
}
public QTimestamp(PathMetadata<?> metadata) {
super(java.sql.Timestamp.class, metadata);
}
public EBoolean period(IDatePeriod period) {
return QueryExtensions.period(this, period);
}
public EBoolean period(ITimestampPeriod period) {
return QueryExtensions.period(this, period);
}
}
</pre><br />
With these generated types all of the property paths in the entity query types of type java.sql.Date and java.sql.Timestamp will use QDate and QTimestamp.<br />
<br />
Now the extensions are ready to use in your queries. In Cyclos 4 the presented extensions are used in many places, for example the CardServiceBean class : <br />
<br />
<pre name="code" class="brush:java">private List<CardVO> doSearch(Class<CardVO> resultType, CardQuery params) {
QCard c = QCard.card;
DBQuery query = from(c);
if (params.getCardNumber() != null) {
query.where(c.cardNumber.eq(params.getCardNumber()));
}
if (params.getCardType() != null) {
query.where(c.cardType().id.eq(params.getCardType().getId()));
}
if (params.getExpirationDate() != null) {
query.where(c.expirationDate.period(params.getExpirationDate()));
}
if (params.getCreationDate() != null) {
query.where(c.creationDate.period(params.getCreationDate()));
}
if (params.getActivationDate() != null) {
query.where(c.activationDate.period(params.getActivationDate()));
}
if (params.getGroups() != null) {
Set<long> groupsIds = EntityVO.getIds(params.getGroups());
query.where(c.owner().group().id.in(groupsIds));
}
if (params.getOwner() != null) {
query.where(c.owner().id.eq(params.getOwner().getId()));
}
if (params.getStatus() != null && !params.getStatus().isEmpty()) {
query.where(c.status.in(params.getStatus()));
}
return query.run(resultType, params, c);
}
</pre><br />
The properties creationDate and activationDate are of type java.sql.Timestamp and expirationDate is of type java.sql.Date.<br />
<br />
<h4>Conclusion</h4><br />
When you find yourself using lots of static helper methods to create your Querydsl queries, then consider using annotated delegate methods instead. They will make your queries more compact and are also easier to write as code autocomplete can be used.Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com3tag:blogger.com,1999:blog-8954277266683132656.post-72919908247462295512010-07-22T21:39:00.002+03:002013-05-08T16:31:32.277+03:00Querying Hibernate with QuerydslQuerydsl provides a typesafe querying layer on top of JPA, JDO, JDBC and other backends. This blog post presents a simple tutorial on how to get started with querying on Hibernate using Querydsl.<br />
<br />
Querydsl for JPA/Hibernate is an alternative to both JPQL and JPA 2 Criteria queries. It combines the dynamic nature of Criteria queries with the expressiveness of JPQL and all that in a fully typesafe manner.<br />
<br />
If you are unsure if switching to Querydsl from JPA 2 Criteria is worth a try, see <a href="http://source.mysema.com/forum/mvnforum/viewthread_thread,49">this comparison</a>.<br />
<br />
<h3>Maven integration</h3><br />
To get started with Querydsl for Hibernate using a Maven 2 based build environment, follow the following steps.<br />
<br />
Add the following dependencies to your Maven project and make sure that the <a href="http://source.mysema.com/maven2/releases">Maven 2 repo of Mysema Source</a> is accessible from your POM :<br />
<br />
<pre name="code" class="brush:xml"><dependency>
<groupId>com.mysema.querydsl</groupId>
<artifactId>querydsl-apt</artifactId>
<version>1.8.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.mysema.querydsl</groupId>
<artifactId>querydsl-hql</artifactId>
<version>1.8.2</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.5.2</version>
</dependency>
</pre><br />
Querydsl uses the Annotation Processing Tool of Java 6 for code generation which needs to be configured as well :<br />
<br />
<pre name="code" class="brush:xml"><project>
<build>
<plugins>
...
<plugin>
<groupId>com.mysema.maven</groupId>
<artifactId>maven-apt-plugin</artifactId>
<version>0.3.2</version>
<executions>
<execution>
<goals>
<goal>process</goal>
</goals>
<configuration>
<outputDirectory>target/generated-sources/java</outputDirectory>
<processor>com.mysema.query.apt.jpa.JPAAnnotationProcessor</processor>
</configuration>
</execution>
</executions>
</plugin>
...
</plugins>
</build>
</project>
</pre><br />
The <i>JPAAnnotationProcessor</i> finds domain types annotated with the <i>javax.persistence.Entity</i> annotation and generates query types for them.<br />
<br />
If you use Hibernate annotations in your domain types you should use the APT processor <i>com.mysema.query.apt.hibernate.HibernateAnnotationProcessor</i> instead.<br />
<br />
Run clean install and you will get your Query types generated into <i>target/generated-sources/java</i>.<br />
<br />
If you use Eclipse, run mvn eclipse:eclipse to update your Eclipse project to include <i>target/generated-sources/java</i> as a source folder.<br />
<br />
Now you are able to construct Querydsl query instances and instances of the query domain model.<br />
<br />
<br />
<h3>Ant integration</h3><br />
If you have an Ant based build environment you can use one of the packaged releases from the <a href="http://source.mysema.com/display/querydsl/Downloads">Downloads page</a>.<br />
<br />
Place the jar files from the full-deps bundle on your classpath and use the following tasks for Querydsl code generation :<br />
<br />
<pre name="code" class="brush:xml"><!-- APT based code generation -->
<javac srcdir="${src}" classpathref="cp">
<compilerarg value="-proc:only"/>
<compilerarg value="-processor"/>
<compilerarg value="com.mysema.query.apt.jpa.JPAAnnotationProcessor"/>
<compilerarg value="-s"/>
<compilerarg value="${generated}"/>
</javac>
<!-- compilation -->
<javac classpathref="cp" destdir="${build}">
<src path="${src}"/>
<src path="${generated}"/>
</javac>
</pre><br />
Replace <i>src</i> with your main source folder, <i>generated</i> with your folder for generated sources and <i>build</i> with your target folder.<br />
<br />
<h3>Using query types</h3><br />
To create queries with Querydsl you need to instantiate variables and Query implementations. We will start with the variables.<br />
<br />
Let's assume that your project has the following domain type :<br />
<br />
<pre name="code" class="brush:java">@Entity
public class Customer {
private String firstName;
private String lastName;
public String getFirstName(){
return firstName;
}
public String getLastName(){
return lastName;
}
public void setFirstName(String fn){
firstName = fn;
}
public void setLastName(String ln)[
lastName = ln;
}
}
</pre><br />
Querydsl will generate a query type with the simple name QCustomer into the same package as Customer. QCustomer can be used as a statically typed variable in Querydsl queries as a representative for the Customer type.<br />
<br />
QCustomer has a default instance variable which can be accessed as a static field :<br />
<br />
<pre name="code" class="brush:java">QCustomer customer = QCustomer.customer;
</pre><br />
Alternatively you can define your own Customer variables like this :<br />
<br />
<pre name="code" class="brush:java">QCustomer customer = new QCustomer("myCustomer");
</pre><br />
<h3>Querying</h3><br />
For the HQL-module HibernateQuery is the main Query implementation. It is instantiated like this :<br />
<br />
<pre name="code" class="brush:java">// where session is a Hibernate session
HQLQuery query = new HibernateQuery (session);
</pre><br />
To use the JPA API instead of the Hibernate API, you can instantiate a JPAQuery like this :<br />
<br />
<pre name="code" class="brush:java">// where entityManager is a JPA EntityManager
HQLQuery query = new JPAQuery (entityManager);
</pre><br />
To retrieve the customer with the first name Bob you would construct a query like this :<br />
<br />
<pre name="code" class="brush:java">QCustomer customer = QCustomer.customer;
HQLQuery query = new HibernateQuery (session);
Customer bob = query.from(customer)
.where(customer.firstName.eq("Bob"))
.uniqueResult(customer);
</pre><br />
The from call defines the query source, the where part defines the filter and uniqueResult defines the projection and tells Querydsl to return a single element. Easy, right?<br />
<br />
To create a query with multiple sources you just use the HQLQuery interface like this :<br />
<br />
<pre name="code" class="brush:java">query.from(customer, company);
</pre><br />
And to use multiple filters use it like this<br />
<br />
<pre name="code" class="brush:java">query.from(customer)
.where(customer.firstName.eq("Bob"), customer.lastName.eq("Wilson"));
</pre><br />
Or like this<br />
<br />
<pre name="code" class="brush:java">query.from(customer)
.where(customer.firstName.eq("Bob").and(customer.lastName.eq("Wilson")));
</pre><br />
In native JPQL form the query would be written like this :<br />
<br />
<pre name="code" class="brush:sql">from Customer as customer
where customer.firstName = "Bob" and customer.lastName = "Wilson"
</pre><br />
<h3>Using joins</h3><br />
Querydsl supports the following join variants in JPQL : inner join, join, left join and full join. Join usage is typesafe, and follows the following pattern :<br />
<br />
<pre name="code" class="brush:java">query.from(cat)
.innerJoin(cat.mate, mate)
.leftJoin(cat.kittens, kitten)
.list(cat);
</pre><br />
The native JPQL version of the query would be<br />
<br />
<pre name="code" class="brush:sql">from Cat as cat
inner join cat.mate as mate
left outer join cat.kittens as kitten
</pre><br />
Another example<br />
<br />
<pre name="code" class="brush:java">query.from(cat)
.leftJoin(cat.kittens, kitten)
.on(kitten.bodyWeight.gt(10.0))
.list(cat);
</pre><br />
With the following JPQL version<br />
<br />
<pre name="code" class="brush:sql">from Cat as cat
left join cat.kittens as kitten
with kitten.bodyWeight > 10.0
</pre><br />
<h3>General usage</h3><br />
Use the the cascading methods of the HQLQuery interface like this<br />
<br />
<ul><li>from : Define the query sources here.</li>
<li>innerJoin, join, leftJoin, fullJoin, on : Define join elements using these constructs. For the join methods the first argument is the join source and the second the target (alias).</li>
<li>where : Define the query filters, either in varargs form separated via commas or cascaded via the and-operator.</li>
<li>groupBy : Define the group by arguments in varargs form.</li>
<li>having : Define the having filter of the "group by" grouping as an varags array of EBoolean expressions.</li>
<li>orderBy : Define the ordering of the result as an varargs array of order expressions. Use asc() and desc() on numeric, string and other comparable expression to access the OrderSpecifier instances.</li>
<li>limit, offset, restrict : Define the paging of the result. Limit for max results, offset for skipping rows and restrict for defining both in one call.</li>
</ul><br />
<h3>Ordering</h3><br />
The syntax for declaring ordering is<br />
<br />
<pre name="code" class="brush:java">query.from(customer)
.orderBy(customer.lastName.asc(), customer.firstName.desc())
.list(customer);
</pre><br />
which is equivalent to the following native JPQL<br />
<br />
<pre name="code" class="brush:sql">from Customer as customer
order by customer.lastName asc, customer.firstName desc
</pre><br />
<h3>Grouping</h3><br />
Grouping can be done in the following form<br />
<br />
<pre name="code" class="brush:java">query.from(customer)
.groupBy(customer.lastName)
.list(customer.lastName);
</pre><br />
which is equivalent to the following native JPQL<br />
<br />
<pre name="code" class="brush:sql">select customer.lastName
from Customer as customer
group by customer.lastName
</pre><br />
<h3>Delete clauses</h3><br />
Delete clauses in Querydsl follow a simple delete-where-execute form. Here are some examples :<br />
<br />
<pre name="code" class="brush:java">QCat cat = QCat.cat;
// delete all cats
new HibernateDeleteClause(session, cat).execute();
// delete all cats with kittens
new HibernateDeleteClause(session, cat).where(cat.kittens.isNotEmpty()).execute();
</pre><br />
The second parameter of the HibernateDeleteClause constructor is the entity to be deleted. The where call is optional and the execute call performs the deletion and returns the amount of deleted entities.<br />
<br />
For JPA based Delete usage, use the JPADeleteClause instead.<br />
<br />
<h3>Update clauses</h3><br />
Update clauses in Querydsl follow a simple update-set/where-execute form. Here are some examples :<br />
<br />
<pre name="code" class="brush:java">QCat cat = QCat.cat;
// rename cats named Bob to Bobby
new HibernateUpdateClause(session, cat).where(cat.name.eq("Bob"))
.set(cat.name, "Bobby")
.execute();
</pre><br />
The second parameter of the HibernateUpdateClause constructor is the entity to be updated. The set invocations define the property updates in SQL-Update-style and the execute call performs the Update and returns the amount of updated entities.<br />
<br />
For JPA based Update usage, use the JPAUpdateClause instead.<br />
<br />
<h3>Subqueries</h3><br />
To create a subquery you create a HQLSubQuery instance, define the query parameters via from, where etc and use unique or list to create a subquery, which is just a type-safe Querydsl expression for the query. unique is used for a unique (single) result and list for a list result.<br />
<br />
<pre name="code" class="brush:java">query.from(department)
.where(department.employees.size().eq(
new HQLSubQuery().from(d).unique(d.employees.size().max())
)).list(department);
</pre><br />
Another example<br />
<br />
<pre name="code" class="brush:java">query.from(employee)
.where(employee.weeklyhours.gt(
new HQLSubQuery().from(employee.department.employees, e)
.where(e.manager.eq(employee.manager))
.unique(e.weeklyhours.avg())
)).list(employee);
</pre><br />
<h3>Exposing the original query</h3><br />
If you need to do tune the original Query before the execution of the query you can expose it like this :<br />
<br />
<pre name="code" class="brush:java">HibernateQuery query = new HibernateQuery(session);
org.hibernate.Query hibQuery = query.from(employee).createQuery(employee);
hibQuery.setResultTransformer(someTransformer);
List results = hibQuery.list();
</pre><br />
<h3>Using Native SQL in Hibernate queries</h3><br />
Querydsl supports Native SQL in Hibernate via the HibernateSQLQuery class.<br />
<br />
To use it, you must generate Querydsl query types for your SQL schema. This can be done for example with the following Maven configuration :<br />
<br />
<pre name="code" class="brush:xml"><plugin>
<groupId>com.mysema.querydsl</groupId>
<artifactId>querydsl-maven-plugin</artifactId>
<version>${project.version}</version>
<executions>
<execution>
<goals>
<goal>export</goal>
</goals>
</execution>
</executions>
<configuration>
<jdbcDriver>org.apache.derby.jdbc.EmbeddedDriver</jdbcDriver>
<jdbcUrl>jdbc:derby:target/demoDB;create=true</jdbcUrl>
<packageName>com.mycompany.mydomain</packageName>
<targetFolder>target/generated-sources/java</targetFolder>
</configuration>
<dependencies>
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derby</artifactId>
<version>${derby.version}</version>
</dependency>
</dependencies>
</plugin>
</pre><br />
When the query types have successfully been generated into the location of your choice, you can use them in your queries.<br />
<br />
Single column query :<br />
<br />
<pre name="code" class="brush:java">// serialization templates
SQLTemplates templates = new DerbyTemplates();
// query types (S* for SQL, Q* for domain types)
SAnimal cat = new SAnimal("cat");
SAnimal mate = new SAnimal("mate");
QCat catEntity = QCat.cat;
HibernateSQLQuery query = new HibernateSQLQuery(session, templates);
List<String> names = query.from(cat).list(cat.name);
</pre><br />
Query multiple columns :<br />
<br />
<pre name="code" class="brush:java">query = new HibernateSQLQuery(session, templates);
List<Object[]> rows = query.from(cat).list(cat.id, cat.name);
</pre><br />
Query all columns :<br />
<br />
<pre name="code" class="brush:java">List<Object[]> rows = query.from(cat).list(cat.all());
</pre><br />
Query in SQL, but project as entity :<br />
<br />
<pre name="code" class="brush:java">query = new HibernateSQLQuery(session, templates);
List<Cat> cats = query.from(cat).orderBy(cat.name.asc()).list(catEntity);
</pre><br />
Query with joins :<br />
<br />
<pre name="code" class="brush:java">query = new HibernateSQLQuery(session, templates);
cats = query.from(cat)
.innerJoin(mate).on(cat.mateId.eq(mate.id))
.where(cat.dtype.eq("Cat"), mate.dtype.eq("Cat"))
.list(catEntity);
</pre><br />
Query and project into DTO :<br />
<br />
<pre name="code" class="brush:java">query = new HibernateSQLQuery(session, templates);
List<CatDTO> catDTOs = query.from(cat)
.orderBy(cat.name.asc())
.list(EConstructor.create(CatDTO.class, cat.id, cat.name));
</pre><br />
If you are using the JPA API instead of the Hibernate API, then use JPASQLQuery instead of HibernateSQLQuery.<br />
<br />
<h3>Where to go next</h3><br />
For an up-to-date guide on how to use Querydsl with Hibernate, see the latest <a href="http://source.mysema.com/display/querydsl/Documentation">Reference Documentation</a>.<br />
<br />
And if you have any issues, questions or ideas for enhancement post a thread in the <a href="http://source.mysema.com/forum/mvnforum/index">Mysema Source Forum</a>.<br />
<br />
Happy querying!Timo Westkämperhttp://www.blogger.com/profile/11246082132389917026noreply@blogger.com26