close
Skip to main content
Springer Nature Link
Log in

Multi-schema-version data management: data independence in the twenty-first century

  • Regular Paper
  • Published:
Save article
View saved research
BERJAYA The VLDB Journal Aims and scope Submit manuscript

Abstract

Agile software development allows us to continuously evolve and run a software system. However, this is not possible in databases, as established methods are very expensive, error-prone, and far from agile. We present InVerDa, a multi-schema-version database management system (MSVDB) for agile database development. MSVDBs realize co-existing schema versions within one database, where each schema version behaves like a regular single-schema database and write operations are propagated between schema versions. Developers use a relationally complete and bidirectional database evolution language (BiDEL) to easily evolve existing schema versions to new ones. BiDEL scripts are more robust, orders of magnitude shorter, and cause only a small performance overhead compared to handwritten SQL scripts. We formally guarantee data independence: no matter how the data of the co-existing schema versions is physically materialized, each schema version is guaranteed to behave like a regular database. Since, the chosen physical materialization significantly determines the overall performance, we equip database administrators with an advisor that proposes an optimized materialization for the current workload, which can improve the performance by orders of magnitude compared to naïve solutions. To our best knowledge, we are the first to facilitate agile evolution of production databases with full support of co-existing schema versions and formally guaranteed data independence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from $39.99 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 2
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 3
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 4
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 5
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 6
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 7
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 8
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 9
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 10
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 11
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 12
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 13
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 14
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 15
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 16
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 17
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 18
BERJAYAThe alternative text for this image may have been generated using AI.
Fig. 19
BERJAYAThe alternative text for this image may have been generated using AI.

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Notes

  1. Online demo available at www.inverda.de.

References

  1. Agrawal, S., Chaudhuri, S., Narasayya, V.R.: Automated selection of materialized views and indexes in SQL databases. In: Proceedings of the 26th International Conference on Very Large Data Bases, 10–14 September 2000, pp. 496–505. Morgan Kaufmann Publishers Inc, San Francisco (2000)

  2. Agrawal, S., Chaudhuri, S., Narasayya, V.: Materialized view and index selection tool for Microsoft SQL server 2000. ACM SIGMOD Rec. 30(2), 608 (2001)

    Article  Google Scholar 

  3. Arora, M., Gosain, A.: Schema evolution for data warehouse: a survey. IJCA 22(5), 6–14 (2011)

    Article  Google Scholar 

  4. Bentley, P., Corne, D.: Creative Evolutionary Systems. Morgan Kaufmann, San Francisco (2002)

    Google Scholar 

  5. Bernstein, P.A., Melnik, S.: Model management 2.0: manipulating richer mappings. In: International Conference on Management of Data (SIGMOD), Beijing, 12–14 June 2007, pp. 1–12 (2007). https://doi.org/10.1145/1247480.1247482

  6. Brodie, M.L., Liu, J.T.: The power and limits of relational technology in the age of information ecosystems. In: On the Move to Meaningful Internet Systems (OTM), Lecture Notes in Computer Science. Springer, Berlin (2010)

  7. Curino, C., Moon, H.J., Deutsch, A., Zaniolo, C.: Automating the database schema evolution process. VLDB J. 22(1), 73–98 (2013)

    Article  Google Scholar 

  8. Curino, C., Moon, H.J., Zaniolo, C.: Graceful database schema evolution: the PRISM workbench. PVLDB 1(1), 761–772 (2008)

    Google Scholar 

  9. Curino, C., Moon, H.J., Tanca, L., Zaniolo, C.: Schema evolution in wikipedia-Toward a web information system benchmark. In: Proceedings of the Tenth International Conference on Enterprise Information Systems (ICEIS), Volume DISI, Barcelona, Spain, 12–16 June 2008, pp. 323–332 (2008)

  10. Domínguez, E., Lloret, J., Rubio, Á.L., Zapata, M.A.: MeDEA: a database evolution architecture with traceability. DKE 65(3), 419–441 (2008)

    Article  Google Scholar 

  11. Floudas, C.A., Pardalos, P.M., Adjiman, C., Esposito, W.R., Gümüs, Z.H., Harding, S.T., Klepeis, J.L., Meyer, C.A., Schweiger, C.A.: Handbook of Test Problems in Local and Global Optimization. Nonconvex Optimization and Its Applications. Springer, New York (2013)

    MATH  Google Scholar 

  12. Gupta, A., Mumick, I.S.: Materialized Views: Techniques, Implementations, and Applications. MIT Press, Cambridge (1999)

    Google Scholar 

  13. Herrmann, K.: Multi-Schema-Version Data Management. Ph.D. thesis, http://nbn-resolving.de/urn:nbn:de:bsz:14-qucosa-231946, Technische Universität Dresden, 2017. Accessed 6 Jun 2018

  14. Herrmann, K., Voigt, Hv , Behrend, A., Lehner, W.: CoDEL—a relationally complete language for database evolution. In: ADBIS (2015)

  15. Herrmann, K., Voigt, H., Behrend, A., Rausch, J., Lehner,W.: Living in parallel realities: Co-existing schema versions with a bidirectional database evolution language. In: Proceedings of the 2017 ACM International Conference on Management of Data (SIGMOD), Chicago, 14–19 May 2017, pp. 1101–1116 (2017). https://doi.org/10.1145/3035918.3064046

  16. Hofmann, M., Pierce, B.C., Wagner, D.: Symmetric lenses. In: Proceedings of the 38th Symposium on Principles of Programming Languages, POPL 2011, Austin, 26–28 January 2011, pp. 371–384 (2011). https://doi.org/10.1145/1926385.1926428

  17. Howard, P.: Data Migration Report, Bloor. https://www.bloorresearch.com/research/data-migration-report-2011/ (2011). Accessed 6 June 2018

  18. McKinna, J.: Complements witness consistency. In: Bx (2016)

  19. Mistry, H., Roy, P., Sudarshan, S., Ramamritham, K.: Materialized view selection and maintenance using multi-query optimization. In: Proceedings of the 2001 international conference on Management of data (SIGMOD), Santa Barbara, CA, USA, 21–24 May 2001, pp. 307–318 (2001). https://doi.org/10.1145/375663.375703

  20. Moon, H.J., Curino, C., Ham, M., Zaniolo, C.: PRIMA—archiving and querying historical data with evolving schemas. In: SIGMOD (2009)

  21. Rahm, E., Bernstein, P.A.: An online bibliography on schema evolution. ACM SIGMOD Rec. 35(4), 30–31 (2006)

    Article  Google Scholar 

  22. Roddick, J.F.: SQL/SE—a query language extension for databases supporting schema evolution. ACM SIGMOD Rec. 21(2), 10–16 (1992)

    Article  Google Scholar 

  23. Roddick, J.F.: A survey of schema versioning issues for database systems. Inf. Softw. Technol. 37(7), 383–393 (1995)

    Article  Google Scholar 

  24. Skoulis, I., Vassiliadis, P., Zarras, A.V.: Open-Source databases: within, outside, or beyond Lehman’s laws of software evolution?. In: Advanced Information Systems Engineering - 26th International Conference, CAiSE 2014, Thessaloniki, Greece, 16–20 June 2014, pp. 379–393 (2014). https://doi.org/10.1007/978-3-319-07881-6_26

  25. Terwilliger, J.F., Cleve, A., Curino, C.: How clean is your sandbox? - Towards a unified theoretical framework for incremental bidirectional transformations In: Theory and Practice of Model Transformations -5th International Conference, ICMT 2012, Prague, Czech Republic, 28–29 May 2012, pp. 1–23 (2012). https://doi.org/10.1007/978-3-642-30476-7_1

  26. Wall, B., Angryk, R.: Minimal data sets vs. synchronized data copies in a schema and data versioning system. In: Proceedings of the 4th workshop on Workshop for Ph.D. students in information and knowledge management (IPKM), Glasgow, United Kingdom, 24–28 October 2011, pp. 67–74 (2011). https://doi.org/10.1145/2065003.2065017

  27. Wentao, W., Chi, Y., Hacígümüş, H., Naughton, J.F.: Towards predicting query execution time for concurrent and dynamic database workloads. PVLDB 6(10), 925–936 (2013)

    Google Scholar 

Download references

Acknowledgements

This work is partly funded by the German Research Foundation (DFG) within the RoSI RTG.

Author information

Authors and Affiliations

  1. Technische Universität Dresden, Dresden, Germany

    Kai Herrmann, Hannes Voigt & Wolfgang Lehner

  2. Aalborg Universitet, Aalborg, Denmark

    Torben Bach Pedersen

Authors
  1. Kai Herrmann
    View author publications

    Search author on:PubMed Google Scholar

  2. Hannes Voigt
    View author publications

    Search author on:PubMed Google Scholar

  3. Torben Bach Pedersen
    View author publications

    Search author on:PubMed Google Scholar

  4. Wolfgang Lehner
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Kai Herrmann.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Herrmann, K., Voigt, H., Pedersen, T.B. et al. Multi-schema-version data management: data independence in the twenty-first century. The VLDB Journal 27, 547–571 (2018). https://doi.org/10.1007/s00778-018-0508-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1007/s00778-018-0508-7

Keywords