Jump to content


Check out our Community Blogs

Register and join over 40,000 other developers!


Recent Status Updates

View All Updates

Photo
* * * * * 1 votes

Who knows how to make a database engine in C?


  • Please log in to reply
22 replies to this topic

#13 BlackRabbit

BlackRabbit

    CodeCall Legend

  • Expert Member
  • PipPipPipPipPipPipPipPip
  • 3871 posts
  • Location:Argentina
  • Programming Language:C, C++, C#, PHP, JavaScript, Transact-SQL, Bash, Others
  • Learning:Java, Others

Posted 09 January 2014 - 09:54 PM

Those look more about using existent database engines than helping coding your own.

 

This 119 pages PDF will help you understand some DB Engine internals you should at least know about when planing to code one.


Edited by BlackRabbit, 09 January 2014 - 09:54 PM.

  • 1

#14 wholegrain

wholegrain

    CC Regular

  • Member
  • PipPipPip
  • 29 posts

Posted 10 January 2014 - 08:10 AM

That's what I thought, but these two seems to go a bit in-depth.

 

Take an in-depth look at the internals of the SQL Server Storage Engine—with advice from a popular author and SQL Server expert. Database developers and administrators get best practices, pragmatic advice, and code samples to help master the intricacies of creating and maintaining enterprise relational databases.



Discover how to:

Upgrade or migrate to SQL Server 2005 and choose configuration options Control space allocation for databases and files manually or automatically Manage transaction logs to maximize efficient restore operations and data consistency Observe the internal structures of clustered and nonclustered indexes Build and partition relational indexes and tables Explore internal storage issues and compare fixed- and variable-length datatypes Detect and correct index fragmentation Implement and manage an appropriate concurrency model using locking or row versioning



    PLUS—Get code samples on the Web.
 

 

Dive deep inside the architecture of SQL Server 2012

Explore the core engine of Microsoft SQL Server 2012—and put that practical knowledge to work. Led by a team of SQL Server experts, you’ll learn the skills you need to exploit key architectural features. Go behind the scenes to understand internal operations for creating, expanding, shrinking, and moving databases—whether you’re a database developer, architect, or administrator.

Discover how to:

  • Dig into SQL Server 2012 architecture and configuration
  • Use the right recovery model and control transaction logging
  • Reduce query execution time through proper index design
  • Track events, from triggers to the Extended Event Engine
  • Examine internal structures with database console commands
  • Transcend row-size limitations with special storage capabilities
  • Choose the right transaction isolation level and concurrency model
  • Take control over query plan caching and reuse

 

But it seems none of them show us the code of the engine itself and explains what they do, and that's what I'd like to have. I have seen good tutorial books for developing game engines and kernels; however, I have yet to find a book on how to develop your own database engine.



 

Edited by wholegrain, 10 January 2014 - 08:35 AM.

  • 0

#15 wholegrain

wholegrain

    CC Regular

  • Member
  • PipPipPip
  • 29 posts

Posted 10 January 2014 - 09:19 AM

http://www.amazon.co..._pr_product_top

 

This books seems the best I have found yet, but it doesn't seem to provide sample code or anything.


  • 0

#16 BlackRabbit

BlackRabbit

    CodeCall Legend

  • Expert Member
  • PipPipPipPipPipPipPipPip
  • 3871 posts
  • Location:Argentina
  • Programming Language:C, C++, C#, PHP, JavaScript, Transact-SQL, Bash, Others
  • Learning:Java, Others

Posted 10 January 2014 - 06:14 PM

What about this? Open source, easy DB Engine, based on old DBF files, DbfDotNet ,  It's a good place to start I would say, I did something like that as one of my first c++ projects.


  • 1

#17 0xDEADBEEF

0xDEADBEEF

    CC Devotee

  • Senior Member
  • PipPipPipPipPipPip
  • 790 posts
  • Programming Language:C, Java, C++, C#, (Visual) Basic, Perl, Transact-SQL, Bash, Prolog, Others
  • Learning:Others

Posted 11 January 2014 - 02:24 AM

A database engine is a very advanced topic. I doubt any books exist with code samples. 

 

I would say that at minimum you've got a few major components. The first is the storage of the tables & indexes, you might want to look into B-Tree structures for storage, simple indexes could be Red-Black trees, or hashmaps. 

 

Next would be the search engine, this would have some sort of 'query plan' which it would then execute. The plan would determine how to positions searches, etc...

 

Then you need to convert SQL to the query plan, so a SQL parsers and query optimiser.


  • 1

Creating SEGFAULTs since 1995.


#18 wholegrain

wholegrain

    CC Regular

  • Member
  • PipPipPip
  • 29 posts

Posted 12 January 2014 - 02:29 PM

What about this? Open source, easy DB Engine, based on old DBF files, DbfDotNet ,  It's a good place to start I would say, I did something like that as one of my first c++ projects.

 

I don't know... Parsing code, I find that even more difficult than writing your own. Do you know like any book I can buy with projects that share certain similarities? It seems if I buy a book on distributed system and search engine, I might be able to actually do this, but I am not quite sure.

 

I want to do several projects that require components that are often used in database engine, before even attempting to write one myself.


A database engine is a very advanced topic. I doubt any books exist with code samples. 

 

I would say that at minimum you've got a few major components. The first is the storage of the tables & indexes, you might want to look into B-Tree structures for storage, simple indexes could be Red-Black trees, or hashmaps. 

 

Next would be the search engine, this would have some sort of 'query plan' which it would then execute. The plan would determine how to positions searches, etc...

 

Then you need to convert SQL to the query plan, so a SQL parsers and query optimiser.

 

Thanks, do you know any small programs I could make that requires a query optimiser and a sql parser, any that requires a b-tree structure, etc?

 

 

Oh, yeah, I found these books. Do you think they may prove to be useful in the long run?

 

http://www.amazon.co...ted programming

 

http://www.amazon.co...ted programming

 

http://www.amazon.co...tributed system

 

http://www.amazon.co...tributed system


Edited by wholegrain, 12 January 2014 - 02:31 PM.

  • 0

#19 0xDEADBEEF

0xDEADBEEF

    CC Devotee

  • Senior Member
  • PipPipPipPipPipPip
  • 790 posts
  • Programming Language:C, Java, C++, C#, (Visual) Basic, Perl, Transact-SQL, Bash, Prolog, Others
  • Learning:Others

Posted 13 January 2014 - 12:04 AM

B-Tree's are just a data structure (http://en.wikipedia.org/wiki/B-tree) so basically if you implement, insert, delete & search on it you'll have the basis of database storage.

 

So you could for example create one of these structures to store a struct with two elements, x&y then once it works expand it out to work with a structure of defined size (i.e. when created you pass a size to it and then all records would be of that size.)

 

SQL Parsing should be easy enough, I'll bet there's already grammars available for the popular parser generators. The query optimizer is a very clever bit of code so its hard to give suggestions where you might start writing such a thing.

 

Not all databases need to run SQL. So you could skip anything todo with sql and provide a lower level access for searching.


  • 0

Creating SEGFAULTs since 1995.


#20 wholegrain

wholegrain

    CC Regular

  • Member
  • PipPipPip
  • 29 posts

Posted 15 January 2014 - 10:16 AM

basic db engine don't need query optimizer, right?

 

basically, i could just store info in a binary file and then implement search, delete, insert and use SQL parsing for the search implementation, and I could call that a basic db engine, right?


  • 0

#21 lespauled

lespauled

    CC Leader

  • Expert Member
  • PipPipPipPipPipPipPip
  • 1360 posts
  • Programming Language:C, C++, C#, JavaScript, PL/SQL, Delphi/Object Pascal, Visual Basic .NET, Pascal, Transact-SQL, Bash

Posted 15 January 2014 - 10:38 AM

You could call a text file a database.  Essentially, that's what DBIII was.


  • 0
My Blog: http://forum.codecal...699-blog-77241/
"Women and Music: I'm always amazed by other people's choices." - David Lee Roth

#22 BlackRabbit

BlackRabbit

    CodeCall Legend

  • Expert Member
  • PipPipPipPipPipPipPipPip
  • 3871 posts
  • Location:Argentina
  • Programming Language:C, C++, C#, PHP, JavaScript, Transact-SQL, Bash, Others
  • Learning:Java, Others

Posted 15 January 2014 - 02:42 PM

basic db engine don't need query optimizer, right?

 

basically, i could just store info in a binary file and then implement search, delete, insert and use SQL parsing for the search implementation, and I could call that a basic db engine, right?

 

Yes, indeed. Of course you forgot the create/drop/alter tables part. Do you have any draft of your current DB project?


  • 0

#23 wholegrain

wholegrain

    CC Regular

  • Member
  • PipPipPip
  • 29 posts

Posted 19 January 2014 - 08:29 PM

Yes, indeed. Of course you forgot the create/drop/alter tables part. Do you have any draft of your current DB project?

 

nah, i don't have a draft.

 

i only begun programming a small database program using swing, but i wanted to try making a db engine at some point in the future.

 

is there any book on how to use SQL parsing libraries or on whatever we need for that?

 

and thanks.


Yes, indeed. Of course you forgot the create/drop/alter tables part. Do you have any draft of your current DB project?

 

creating, dropping and altering table from the SQL commands should be kinda hard. Is there any book or tutorial for that?

 

i thought about making an arraylist containing instances of a certain class, but I need to dynamically create new classes if I were to implement the create table function, and it wouldn't be a table object, but rather an arraylist that's used to fill up a table from the Swing library.


Edited by wholegrain, 19 January 2014 - 08:34 PM.

  • 0




Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download