Shantanu's Blog

Corporate Consultant

January 31, 2008

 

Find redundant indexes and import data in InnoDB

How do you find probably useless indexes?
here is a query to find all the indexes in the database "my_database" that have a cardinality/count ratio of less than 0.1% Such indexes are probably not very useful, and should be looked at carefully to justify their existence.

use information_schema;
select tables.table_name, statistics.index_name, statistics.cardinality, tables.table_rows
from tables
join statistics
on (statistics.table_name = tables.table_name
and statistics.table_schema = 'SBI_IPO_CBS'
and ((tables.table_rows / statistics.cardinality) > 1000))

How do I find out the tables those are occupying maximum space on the disk?

SELECT concat(table_schema,'.',table_name),
table_rows,concat(round(data_length/(1024*1024*1024),2),'G') DATA,
concat(round(index_length/(1024*1024*1024),2),'G')
idx,concat(round((data_length+index_length)/(1024*1024*1024),2),'G')
total_size,round(index_length/data_length,2) idxfrac
FROM TABLES



http://fallenpegasus.livejournal.com/678743.html



To import 100 million rows, use the Unix/Linux 'split' command to convert the file with 100 million rows into 100 files of one million rows.
Next run a bash script to actually pull in each file. Here it is:

#!/bin/bash
for i in $( ls -1 /data/tmp ); do
mysql -e "load data infile '/data/tmp/$i' into table db_name.table.name;"
done


http://www.paragon-cs.com/wordpress/?p=81

Labels:


January 29, 2008

 

php functions, tips and syntax

Phil Thompson has written about "7 PHP functions that saved his life".


http://imgiseverything.co.uk/2008/01/18/7-php-functions-that-saved-my-life/


I found the tips in the comments more valuable than the article!

1) You know you can combine the echo and the exit() call:
exit(”x is greater than y”);
When debugging forms, I find it very handy to write a little helper function that spits out the contents of the $_POST array wrapped in PRE tags, then call that:
die(post_contents());

2) The substr() is often not needed if you only need the first or the nth character, you can use the curly-braces syntax in those situations:
For example: substr($var, 0, 1) is the same as $var{0}

3) // instead of this:
if(isset($variable) && $variable != ”){}
// you can do this:
if(!empty($variable)){}
// it’s the same

4) switching to python from php was how i really saved my life. Try that language and its frameworks, don’t die on php! :P

Labels:


 

Keys and locks!

How do I know which tables are having primary or other indexes and which tables do not have any indexes?

Ans:

All the tables with or without indexes showing deails of constraint name and column on which the index is built.

select CONCAT(t.table_name,".",t.table_schema) as tbl, c.column_name,c.constraint_name
from TABLES AS t LEFT JOIN KEY_COLUMN_USAGE AS c
ON (t.TABLE_NAME=c.TABLE_NAME AND c.CONSTRAINT_SCHEMA=t.TABLE_SCHEMA AND constraint_name="PRIMARY")
WHERE t.table_schema!="information_schema"
order by constraint_name;

If you add AND constraint_name IS NULL to the where clause, you will find all the tables without any key!

select CONCAT(t.table_name,".",t.table_schema) as tbl
from TABLES AS t LEFT JOIN KEY_COLUMN_USAGE AS c
ON (t.TABLE_NAME=c.TABLE_NAME AND c.CONSTRAINT_SCHEMA=t.TABLE_SCHEMA AND constraint_name='PRIMARY')
WHERE t.table_schema!="information_schema" AND constraint_name IS NULL;

If you want to know the names of the tables thouse have primary key, change the left join in the first query to INNER JOIN.

select CONCAT(t.table_name,".",t.table_schema) as tbl, c.column_name,c.constraint_name
from TABLES AS t INNER JOIN KEY_COLUMN_USAGE AS c
ON (t.TABLE_NAME=c.TABLE_NAME AND c.CONSTRAINT_SCHEMA=t.TABLE_SCHEMA AND constraint_name="PRIMARY")
WHERE t.table_schema!="information_schema"
order by constraint_name;

You need to use "information_schema" database in order to run these queries!!

Labels:


 

MySQL Case Study - 165

I have a column named "path" containing web urls and I'd like to extract the file extension from those that have one (some are folder links) and place it

into a column named "ext" for quick sorting of the content and I was wondering how I'd construct a query to do that for me. In the future, I'll have my PHP

script do it as the urls are added but for now I have a quite a bit of legacy content to sort through.

Ans:

select url,
lower(
substring_index(
substring_index(
case
when locate('.',substring_index(url, '/', -1)) <> 0 then substring_index(url, '/', -1)
else '.'
end,
'.', -1),
'?', 1)
) from yourtable

Hope this helps :)

obviously, you need to convert it into an update statement,

http://forums.mysql.com/read.php?10,190785,190800#msg-190800

_____


I have to implement something which apparently looks pretty simple, but in fact it is not.

Let assume the table : MYTABLE with 2 columns : A and B

For example :

A B
10 20
12 14
12 20
15 20
10 30
12 30
18 40
15 25

I'd like to find values of B where A=10 and A=12 AND A=15 for the same value of B. In my example it would return only B=20.

Is there a simple way of doing this ? I tried using subqueries, but I failed.

Ans:

SELECT b FROM mytable
WHERE a=10 OR a=12 OR a = 15
GROUP BY b
HAVING COUNT(a) = 3
HAVING COUNT( DISTINCT a ) = 3;

http://forums.mysql.com/read.php?10,190709,190720#msg-190720

_____

I have been playing with this query for a while and can't get the results i want. can somone help me.

here is my Query:

SELECT (ps.round1+ps.handicap) AS rd1
FROM userchoice uc
JOIN players p
ON uc.playerId=p.playerId
JOIN playerscores ps
ON ps.playerId=uc.playerId AND ps.tournamentId=uc.tourneyId AND ps.round1!='0'
WHERE uc.username='billy' AND uc.entryId=1
ORDER BY rd1 ASC
LIMIT 4

Results:
Rd1
60
65
68
69

Now, how can i modify this query so that it adds those 4 numbers together and returns the total from the four (which in this case would be 262)

I tried throwing a SUM(ps.round1+ps.handicap) AS rd1 in there, but that gave me a value of 497.

Ans:

SELECT SUM(rd1)
FROM (
SELECT ps.round1+ps.handicap AS rd1
FROM userchoice uc
JOIN players p
ON uc.playerId=p.playerId
JOIN playerscores ps ON ps.playerId=uc.playerId AND ps.tournamentId=uc.tourneyId AND ps.round1!='0'
WHERE uc.username='billy' AND uc.entryId=1
ORDER BY rd1 ASC LIMIT 4
) AS tmp;


http://forums.mysql.com/read.php?10,190447,190458#msg-190458

_____

Can someone please help me with an sql query to filter duplicated rows in a table. The problem is that sometimes row1 and row2 fields are reversed but this is something that i would like removed from the table.

name1 | name2
-----------------
John | Jane
Fred | Claudia
Jim | Claire
Jane | John
Claire | Jim

So if rows where name1/name2=name2/name1 then one of these rows should be filtered. i.e if John/Jane appears on one row then Jane/john should be removed from the table in the other row.

Sounds easy but i cannot find a way to do it, any ideas gurus out there?

Ans:

DELETE n1.* FROM names AS n1,names AS n2 WHERE n1.name1 = n2.name2 AND n2.name1 = n1.name2;


http://forums.mysql.com/read.php?10,190338,190345#msg-190345

_____

Other notable threads are as follows:
1) MySQL interview questions...

http://forums.mysql.com/read.php?10,191044,191044#msg-191044

2) Encrypted passwords
http://forums.mysql.com/read.php?10,190993,190993#msg-190993

3) Full Text Search
http://forums.mysql.com/read.php?10,190979,190979#msg-190979

http://forums.mysql.com/read.php?10,193470,193470#msg-193470

4) Subquery Optimization

http://forums.mysql.com/read.php?10,192870,192891#msg-192891

http://forums.mysql.com/read.php?10,192503,192511#msg-192511

Labels:


 

MySQL Case Study - 164

Records 2 days old

I'm trying to display active records in a table on my web page. However, to minimize maintenance, I only want to display records that are up to 48 hours old at any point. Here's my code so far:

$query="SELECT * from birthdays where Active='1' AND Event='request' AND hour(DateEntered)< 48";

Where "Active" is a flag at 1/0..."Event" is populated and "DateEntered" is a system TimeStamp of the record creation.

I have 11 Active records and 5 that are 48 hours old, or less, so my page should return 5 records. It's returning all 11.

Any help?

http://forums.mysql.com/read.php?10,190351,190351#msg-190351

Labels:


 

MySQL Case Study - 163

retrieve enum values
There is a column of type enum ( 'A', 'B', 'C', 'D').
How can I get via a query all enum values available for column ?

Is there a better query than:
describe table column;

I would prefer a query which give responses like:
"A"
"B"
"C"
"D"

http://forums.mysql.com/read.php?10,190408,190408#msg-190408

Labels:


 

MySQL Case Study - 162

Calc daily total amount for a specified time period
I have not had a lot of experience with MySql and could use some help. I have a table called deposits with an id, username, amount and created_at. I have a form that allows a user to select a start at and stop at date for the search. What I want to do is select the daily total amount for each day within the selected fields.
For example get the daily totals from Jan 1st 08 to Jan 7th 08.
I know I can get a specific day with
SELECT SUM(amount) As Total
FROM deposits
WHERE created_at > '2007-11-11';
But I can not figure out how to do this for more then one day.

http://forums.mysql.com/read.php?10,190787,190787#msg-190787

Labels:


 

MySQL Case Study - 161

SELECT grouping problem with MAX()

I've been ripping my hair out trying to get this select to return what I want. Here is what I have so far:-

create table paylist (date int(8), username varchar(12), amount real(12,2))

Some example data:-
INSERT into paylist(date, amount, username) values ('20080101', 10, 'bob'), ('20080102', 20, 'bob'), ('20080103', 5, 'bob'), ('20080102, 0, 'laura')

What I want is to get returned is this:-
20080103, 5, bob
20080102, 0, laura

The real table has lots of different users. Now if I use:-

SELECT MAX( date ) AS m, amount, username
FROM `paylist`
WHERE 1
GROUP BY username
ORDER BY m DESC

I Get:-
20080103, 10, bob
20080102, 0, laura

Seems no matter what I try, I cannot get the amount that corrosponds with the MAX( date ).

http://forums.mysql.com/read.php?10,192556,192556#msg-192556

Labels:


 

MySQL Case Study - 160

Doing calculations in linked tables

I have two tables, one with assets and one with tasks. Each asset can have n number of tasks assigned to it and each task has a status number/percent. What I want to do is to create a view that contains the average status of all tasks assigned to an asset, and join that with the asset list.

Maybe this exlpains it better. Here are the tables I have, and the resulting view.

Asset table:
id name
1 asset1
2 asset2

Task table:
id name status linked_asset_id
1 task1 20 1
2 task2 30 1
3 task 5 2

Resulting view (joining the asset and view should be)
id name status
1 asset1 25
2 asset2 5

http://forums.mysql.com/read.php?10,193432,193432#msg-193432

Labels:


 

MySQL Case Study - 159

GROUP BY, MIN() row ordering problems

I am currently putting together a sports results/rankings database web application and are experiencing problems with the 'GROUP BY' MIN() function.

The two tables I am referring to:

TABLE 1 = athletes
athleteID (PK), nameFirst, nameLast, gender, DOB, club

TABLE 2 = results
resultID (PK), athleteID, event, ageGroup, raceTime, placing, competition, date

Some simple sample data for TABLE 1 (results):
resultID, athleteID, event, raceTime, placing, competition, date
1 | 182 | 100m | 10.50 | 1 | League | 2008-12-12
2 | 12 | 100m | 10.60 | 2 | Nationals | 2008-11-15
3 | 268 | 100m | 10.40 | 1 | League | 2008-06-05
4 | 9 | 100m | 11.20 | 4 | Local | 2008-07-03
5 | 12 | 100m | 11.60 | 3 | League | 2008-12-01
6 | 182 | 100m | 10.10 | 1 | Challenge | 2008-10-19


I wish to create a query that will return the fastest time for each athlete, and return only one unique result for each athlete (athleteID).

So far I have this:

SELECT *, MIN(raceTime)
FROM athletes, results
WHERE results.athleteID = athletes.athleteID
GROUP BY resultID
ORDER BY raceTime ASC

This works great, but it returns more than one record for each athlete:

resultID, athleteID, event, raceTime, placing, competition, date
6 | 182 | 100m | 10.10 | 1 | Challenge | 2008-10-19
3 | 268 | 100m | 10.40 | 1 | League | 2008-06-05
1 | 182 | 100m | 10.50 | 1 | League | 2008-12-12
2 | 12 | 100m | 10.60 | 2 | Nationals | 2008-11-15
4 | 9 | 100m | 11.20 | 4 | Local | 2008-07-03
5 | 12 | 100m | 11.60 | 3 | League | 2008-12-01

I am after this result set (fastest raceTime for unique athleteID):

resultID, athleteID, event, raceTime, placing, competition, date
6 | 182 | 100m | 10.10 | 1 | Challenge | 2008-10-19
3 | 268 | 100m | 10.40 | 1 | League | 2008-06-05
2 | 12 | 100m | 10.60 | 2 | Nationals | 2008-11-15
4 | 9 | 100m | 11.20 | 4 | Local | 2008-07-03

I also tried this:

SELECT *, MIN(raceTime)
FROM athletes, results
WHERE athletes.athleteID = results.athleteID
GROUP BY results.athleteID
ORDER BY raceTime ASC

This produces the right athleteID with the right raceTime, but the other fields are showing incorrect values:

resultID, athleteID, event, raceTime, placing, competition, date
6 | 182 | 100m | 10.10 | 1 | League | 2008-12-12
3 | 268 | 100m | 10.40 | 1 | League | 2008-06-05
2 | 12 | 100m | 10.60 | 3 | League | 2008-12-01
4 | 9 | 100m | 11.20 | 4 | Local | 2008-07-03

Any suggestions would be graetefully accepted.

http://forums.mysql.com/read.php?10,193658,193658#msg-193658

 

MySQL Case Study - 158

Finding a way to omit certain letters

I have a table with a field called CELL under this field I have rows with the following.

1X3V
1Y3V
1Z3V
.
.
.
.
.
up to
.
.
.
30X3V
30Y3V
30Z3v

I would like to find a function to omit the (X,Y,Z)3V and only output 1 - 30.

Can this be done?

http://forums.mysql.com/read.php?10,192708,192708#msg-192708

Labels:


 

MySQL Case Study - 157

reference a column in the following recordset
Let 'playlist' be a sequence of images in a specified order. The items of the playlists are stored in a table like the following:

CREATE TABLE `playlist_items` (
`id` smallint(6) NOT NULL auto_increment,
`list_id` smallint(6) NOT NULL,
`display_id` smallint(6) NOT NULL,
`fs_path` text NOT NULL,
PRIMARY KEY (`id`),
KEY `display_index` (`list_id`,`display_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8

After the following query

SELECT display_id,fs_path FROM playlist_items WHERE list_id=123 AND display_id>=12 LIMIT 10;

I receive a result set like

+------------+-----------------------+
| display_id | fs_path |
+------------+-----------------------+
| 12 | /path/to/img12.jpg |
| 13 | /path/to/img13.jpg |
| 14 | /path/to/img14.jpg |
| 15 | /path/to/img15.jpg |
| 16 | /path/to/img16.jpg |
| 17 | /path/to/img17.jpg |
| 18 | /path/to/img18.jpg |
| 19 | /path/to/img19.jpg |
| 20 | /path/to/img20.jpg |
| 21 | /path/to/img21.jpg |
+------------+-----------------------+

Is there any means in SQL to specify a SQL query the returns the following result set from the table defined above?

+------------+-----------------------+-----------------------+
| display_id | fs_path | next |
+------------+-----------------------+-----------------------+
| 12 | /path/to/img12.jpg | /path/to/img13.jpg |
| 13 | /path/to/img13.jpg | /path/to/img14.jpg |
| 14 | /path/to/img14.jpg | /path/to/img15.jpg |
| 15 | /path/to/img15.jpg | /path/to/img16.jpg |
| 16 | /path/to/img16.jpg | /path/to/img17.jpg |
| 17 | /path/to/img17.jpg | /path/to/img18.jpg |
| 18 | /path/to/img18.jpg | /path/to/img19.jpg |
| 19 | /path/to/img19.jpg | /path/to/img20.jpg |
| 20 | /path/to/img20.jpg | /path/to/img21.jpg |
| 21 | /path/to/img21.jpg | /path/to/img22.jpg |
+------------+-----------------------+-----------------------+

Or in other words, are there means to reference a column in the recordset following a recordset?

I was thinking about that and did not find a solution which will solve the problem with using SQL only.

http://forums.mysql.com/read.php?10,191547,191547#msg-191547

Labels:


January 28, 2008

 

MySQL Case Study - 156

Nested Select using Like

I am trying to use the results of a nested select statement in the 'parent' statement using 'LIKE' as per:

SELECT * FROM t1 WHERE c1 LIKE (SELECT c7 FROM t2 WHERE ID = 987);

Does anyone know if this is possible? I am getting empty sets when I experiment with the placement of the % signs. For example:

SELECT * FROM t1 WHERE c1 LIKE "%(SELECT c7 FROM t2 WHERE ID = 987)%";
Empty set (0.01 sec)

Also I have tried to assign the result of the nested select to a variable like:
SELECT @v := c7 FROM t2 WHERE ID = 987;

But again then have no idea how to put this variable "@v" into a statement using LIKE.

Have tried:
SELECT * FROM t1 WHERE c1 LIKE '%@v%';
Empty set (0.00 sec)

Thanks for any help.

http://forums.mysql.com/read.php?10,191287,191287#msg-191287

Labels:


 

MySQL Case Study - 155

Updates in independent and dependent subqueries
I have this query:

SELECT member_num, COUNT(member_num) FROM members GROUP BY member_num;

It works and gives me exactly what I want it to give me. Now what I need to do is take the value gathered by COUNT(member_num) and update a column on another table with that value.

I've been doing some research, and I know SELECT has a "FOR UPDATE" clause that I think might work. The problem I run into comes with the UPDATE statement that follows the SELECT statement. How do I get the value from COUNT to work in an update statement?

The best I could come up with is something like this:

SELECT member_num, COUNT(member_num) AS num_members FROM members GROUP BY member_num FOR UPDATE;
UPDATE member_summary SET quantity = num_members WHERE members.member_num = member_summary.member_num;

It's convoluted and doesn't even work. But I don't know what else to do to make it work. My scripting skills are sub-par at best.

http://community.livejournal.com/mysql/123769.html

Labels:


January 13, 2008

 

Download site for offline browsing

How do you download the entire site for offline browser?
Download the HTTrack software from the site...
http://www.httrack.com/page/2/en/index.html

And follow the steps given below...

1) Type the new Project name for e.g. upakram
Decide the directory in which you want to save the downloaded files.


2) Type the Web Address for e.g. http://mr.upakram.org/
Click on set options... button



3) use Maximum mirroring depth as 1 and maximum external depth as 0 if you don't want external files to be cached.
It means if a page links to yahoo.com HTTrack will not download the yahoo.com page.
You can change this value to 2 or 3 if you want 2 internal links on yahoo.com to be downloaded to your hard drive.


You can now create a zip file something like this...

http://saraswaticlasses.net/manogat/upakram.zip

Enjoy offline browsing!

Archives

June 2001   July 2001   January 2003   May 2003   September 2003   October 2003   December 2003   January 2004   February 2004   March 2004   April 2004   May 2004   June 2004   July 2004   August 2004   September 2004   October 2004   November 2004   December 2004   January 2005   February 2005   March 2005   April 2005   May 2005   June 2005   July 2005   August 2005   September 2005   October 2005   November 2005   December 2005   January 2006   February 2006   March 2006   April 2006   May 2006   June 2006   July 2006   August 2006   September 2006   October 2006   November 2006   December 2006   January 2007   February 2007   March 2007   April 2007   June 2007   July 2007   August 2007   September 2007   October 2007   November 2007   December 2007   January 2008   February 2008   March 2008   April 2008   July 2008   August 2008   September 2008   October 2008   November 2008   December 2008   January 2009   February 2009   March 2009   April 2009   May 2009   June 2009   July 2009   August 2009   September 2009   October 2009   November 2009   December 2009   January 2010   February 2010   March 2010   April 2010   May 2010   June 2010   July 2010   August 2010   September 2010   October 2010   November 2010   December 2010   January 2011   February 2011   March 2011   April 2011   May 2011   June 2011   July 2011   August 2011   September 2011   October 2011   November 2011   December 2011   January 2012   February 2012   March 2012   April 2012   May 2012   June 2012   July 2012   August 2012   October 2012   November 2012   December 2012   January 2013   February 2013   March 2013   April 2013   May 2013   June 2013   July 2013   September 2013   October 2013   January 2014   March 2014   April 2014   May 2014   July 2014   August 2014   September 2014   October 2014   November 2014   December 2014   January 2015   February 2015   March 2015   April 2015   May 2015   June 2015   July 2015   August 2015   September 2015   January 2016   February 2016   March 2016   April 2016   May 2016   June 2016   July 2016   August 2016   September 2016   October 2016   November 2016   December 2016   January 2017   February 2017   April 2017   May 2017   June 2017   July 2017   August 2017   September 2017   October 2017   November 2017  

This page is powered by Blogger. Isn't yours?